Monthly Archives: November 2013

Release of Smap Data Cleansing

Announcing the release of Smap Evaluation Tools Server version 13.11. This version has concentrated on some data cleansing functionality which as been requested by a team assessing the needs of people affected by Typhoon Haiyan.  The Speed Evidence project has provided funding.

New functionality

  1. Updating data on the server prior to export. (Data Cleansing).
  2. Ability to hide the graph legend as this sometimes obscures the data. http://blog.smap.com.au/hiding-gaph-labels/
  3. Bundling of Smap Uploader
  4. Support for Apache 2.4

Data Cleansing

The following video shows how to update text values on the server.

The analyst training material has also been updated to included more detailed instructions data cleansing.

Other changes

A bug that prevented the creation of graphs when question names included dashes has been fixed.

Uploading of Polygons and Lines has been made more robust.  Previously if a valid Polygon had not been recorded (more than 2 points) then the survey was not uploaded.  Now the rest of the data is uploaded and the invalid Polygon / Line is discarded.

Availability

A downloadable tar file of the updated server should be available tomorrow.  Hosted servers will be progressively updated over the next few days.

Limits on the number of questions per form

Recently a Smap customer, who are conducting a hospital audit, encountered problems loading a survey that had just under 2,000 questions.  The problem was caused by a limit in the Postgres database. There can be no more than 1,600 columns per table.  In a survey you can reach this limit either by specifying:

  • 1,600 questions
  • or 160 multiple choice questions with 10 options in each, since a separate column is used for each option in a multiple choice question
  • or a combination of the above

The solution to the problem is to enclose some questions within a repeat group with a “repeat_count” of “1”.  These questions will be written to a separate table in the database.

While the data collector is filling in the survey they will not be aware of this repeat group as it is set to automatically repeat just once.

When you export the survey results, if you select the “pivot” option, then the records from this “begin repeat” subform will be combined with the records from the main form.  However you will also have the option of exporting the main form and its sub form(s) separately.  I’ve seen exports of surveys to spreadsheets with over 3,000 columns and I’m not aware of any constraints on this.

I was able to load the large hospital audit survey after adding this begin repeat, end repeat solution. A future release of Smap will check that the maximum number of questions per form is not exceeded. If it is an error message will be shown.

Hiding Gaph Labels

I have released a patch to the Smap Analysis software in response to a request from the team in the Philippines who are responding to Typhoon Hainan.  This allows you to hide labels on a graph by clicking on an icon labelled “L” at the top right of the panel.  Clicking on the “L” again will show the labels.  This is available in the dashboard and in reports.

graph_no_label

Graph with hidden labels. Note the “L” icon at top right.

When the labels are hidden you can see what each of the bars represent by moving your mouse over the bar.  This will show the label just for that bar along with the bar’s value.

This patch to the server also fixes a bug so that the description added by the analyst is now shown on the report.

This release is only available on a limited number of the hosted servers at the moment but will be included in the next monthly release.

Smap Uploader

The Smap Uploader tool is now available.  This will be bundled with the server in the next release of Smap.

Ideally you should never need to use Smap Uploader!  It is much more straightforward to connect the phones directly to the server over a network.  However you may have backed up your surveys and then the phone has been damaged, or the uploads from that phone may not work for some reason. In these cases Smap Uploader provides a fall back.

Click on the Smap Uploader button on http://dev.smap.com.au to download. You will need Java 6 or later https://java.com/en/download/index.jsp.  Save the downloaded file somewhere where you can find it again, possibly on your desktop.  Double click to start it.

su

Smap Uploader Screen

 

You can copy the forms from your phone using a USB cable.  The exact process may vary depending on the phone but commonly you will need to attach as a USB mass storage device. The forms can be founds under /sdcard/fieldTask/instances.  Each one will have its own directory where attachments are also stored.

In Smap Uploader choose the above folder and fill in the server name, your user id and your password. Then press submit. Progress will be shown in the status panel along with an hourglass.

If you have to stop sending part way through you can just exit Smap Uploader.  It will resume from the point you left off next time you select the form folder and press submit.

You may end up sending forms multiple times.  For example some forms could have been sent from the phone and then also sent from Smap Uploader.  In this case the duplicate forms will be discarded by the server.

Release of Smap Shapefiles Export

Announcing the release Smap Evaluation Tools Server version 13.10. New functionality includes:

  1. Exporting surveys to ESRI shapefiles.
  2. Administration and analysis screens are a bit more mobile phone friendly.  With the this release you should be able to create and view analysis panels on your mobile phone now.  However this is work in progress.
  3. Surveys that have been blocked will no longer be available for download.  Submission of results from these surveys will continue to be prevented until the survey is un-blocked.
  4. Incomplete surveys are no longer included in results automatically.  These can be uploaded from the phones but they will be marked as “bad” surveys with a reason of “incomplete” by default.

Exporting Shapefiles

Individual forms within a survey can be exported as shapefiles and then imported into a GIS such as ESRI.

The Smap Server now supports exporting to XLS, CSV, Openstreetmap as well as shapefiles.

Blocking Surveys

Installing

You can download the server tar file from here.