Author Archives: Neil Penman

Time Zones in Reports and Graphs

Leave a reply

Its timely to look at the use of time zones in reports and graphs since we will shortly be releasing a version of Smap that includes trend analysis.  This will allow you to create graphs and maps  showing trends in data against time.

The most common date columns in Smap reports are the start and end of the survey.  When these are shown in a table on the Smap server the time zone is included. For example the following is from a survey that started at 7:10pm in a time zone GMT + 11.

  • 2012-03-13 19:10:27.412+11

Note the time zone is shown as +11. When the survey is exported to Excel the time zone information is removed and the time is shown as GMT.

  • 13/03/2012  8:10:27 AM

This is because Excel can’t include the time zone in a date cell.  (At least I don’t think it can). However you can modify the time back to local time.  Dates in Excel are stored as floating point numbers.  If you add “1” to this date it will increment by 1 day.  So to change the time to a time zone of GMT + 11 you can add 11/24.

In a future release of Smap I will add an extra column for the time zone of the survey so that this information is available in the report.

Now getting back to the trend analysis. The times in the x-axis of trend graphs will be shown in the local time of the survey without timezone information.  So for the example above where the time interval is set to show results by the hour.  That survey result will fall within the x-axis period of:

  • 2012-03-13 19

In summary all dates and times in reports are in local time except when exported to Excel when they are shown as GMT.

Release of Smap Data Cleansing

3 Replies

Announcing the release of Smap Evaluation Tools Server version 13.11. This version has concentrated on some data cleansing functionality which as been requested by a team assessing the needs of people affected by Typhoon Haiyan.  The Speed Evidence project has provided funding.

New functionality

  1. Updating data on the server prior to export. (Data Cleansing).
  2. Ability to hide the graph legend as this sometimes obscures the data. http://blog.zarkman.eu/hiding-gaph-labels/
  3. Bundling of Smap Uploader
  4. Support for Apache 2.4

Data Cleansing

The following video shows how to update text values on the server.

The analyst training material has also been updated to included more detailed instructions data cleansing.

Other changes

A bug that prevented the creation of graphs when question names included dashes has been fixed.

Uploading of Polygons and Lines has been made more robust.  Previously if a valid Polygon had not been recorded (more than 2 points) then the survey was not uploaded.  Now the rest of the data is uploaded and the invalid Polygon / Line is discarded.

Availability

A downloadable tar file of the updated server should be available tomorrow.  Hosted servers will be progressively updated over the next few days.

Limits on the number of questions per form

1 Reply

Recently a Smap customer, who are conducting a hospital audit, encountered problems loading a survey that had just under 2,000 questions.  The problem was caused by a limit in the Postgres database. There can be no more than 1,600 columns per table.  In a survey you can reach this limit either by specifying:

  • 1,600 questions
  • or 160 multiple choice questions with 10 options in each, since a separate column is used for each option in a multiple choice question
  • or a combination of the above

The solution to the problem is to enclose some questions within a repeat group with a “repeat_count” of “1”.  These questions will be written to a separate table in the database.

While the data collector is filling in the survey they will not be aware of this repeat group as it is set to automatically repeat just once.

When you export the survey results, if you select the “pivot” option, then the records from this “begin repeat” subform will be combined with the records from the main form.  However you will also have the option of exporting the main form and its sub form(s) separately.  I’ve seen exports of surveys to spreadsheets with over 3,000 columns and I’m not aware of any constraints on this.

I was able to load the large hospital audit survey after adding this begin repeat, end repeat solution. A future release of Smap will check that the maximum number of questions per form is not exceeded. If it is an error message will be shown.

Hiding Gaph Labels

Leave a reply

I have released a patch to the Smap Analysis software in response to a request from the team in the Philippines who are responding to Typhoon Hainan.  This allows you to hide labels on a graph by clicking on an icon labelled “L” at the top right of the panel.  Clicking on the “L” again will show the labels.  This is available in the dashboard and in reports.

graph_no_label

Graph with hidden labels. Note the “L” icon at top right.

When the labels are hidden you can see what each of the bars represent by moving your mouse over the bar.  This will show the label just for that bar along with the bar’s value.

This patch to the server also fixes a bug so that the description added by the analyst is now shown on the report.

This release is only available on a limited number of the hosted servers at the moment but will be included in the next monthly release.

Smap Uploader

Leave a reply

The Smap Uploader tool is now available.  This will be bundled with the server in the next release of Smap.

Ideally you should never need to use Smap Uploader!  It is much more straightforward to connect the phones directly to the server over a network.  However you may have backed up your surveys and then the phone has been damaged, or the uploads from that phone may not work for some reason. In these cases Smap Uploader provides a fall back.

Click on the Smap Uploader button on http://dev.smap.com.au to download. You will need Java 6 or later https://java.com/en/download/index.jsp.  Save the downloaded file somewhere where you can find it again, possibly on your desktop.  Double click to start it.

su

Smap Uploader Screen

 

You can copy the forms from your phone using a USB cable.  The exact process may vary depending on the phone but commonly you will need to attach as a USB mass storage device. The forms can be founds under /sdcard/fieldTask/instances.  Each one will have its own directory where attachments are also stored.

In Smap Uploader choose the above folder and fill in the server name, your user id and your password. Then press submit. Progress will be shown in the status panel along with an hourglass.

If you have to stop sending part way through you can just exit Smap Uploader.  It will resume from the point you left off next time you select the form folder and press submit.

You may end up sending forms multiple times.  For example some forms could have been sent from the phone and then also sent from Smap Uploader.  In this case the duplicate forms will be discarded by the server.

Release of Smap Shapefiles Export

Leave a reply

Announcing the release Smap Evaluation Tools Server version 13.10. New functionality includes:

  1. Exporting surveys to ESRI shapefiles.
  2. Administration and analysis screens are a bit more mobile phone friendly.  With the this release you should be able to create and view analysis panels on your mobile phone now.  However this is work in progress.
  3. Surveys that have been blocked will no longer be available for download.  Submission of results from these surveys will continue to be prevented until the survey is un-blocked.
  4. Incomplete surveys are no longer included in results automatically.  These can be uploaded from the phones but they will be marked as “bad” surveys with a reason of “incomplete” by default.

Exporting Shapefiles

Individual forms within a survey can be exported as shapefiles and then imported into a GIS such as ESRI.

The Smap Server now supports exporting to XLS, CSV, Openstreetmap as well as shapefiles.

Blocking Surveys

Installing

You can download the server tar file from here.

 

Assessment Reports for Speed Evidence

2 Replies

The Speed Evidence project has released a video showing the features of the Information Management Portal. This is looking really good.

However up until today media files such as pictures that were included in a table of data or a map were not being copied across into the report shown by the portal. The Speed Evidence portal pulls assessment reports from the Smap Server using the reports API. These reports consist of a graph, map, table, picture, video or audio file that have been collected using the Smap Mobile Phone tools.   The report contents have to be copied into a report location on the server so that they can be accessed using the report identifier rather than a user name and password.  This report identifier will look something like this “293ffa7b-70j7-4dc0-gb4a-57fcb45a20a3” and uniquely identifies the report.

This issue has now been fixed. When a report is created now all the media files that are included in tables and maps are copied over to the reports area and assigned the report identifier. You can see this in the example report below which is a table of data containing pictures grouped by the nationality of the respondent.

https://dev.smap.com.au/surveyKPI/reports/view/ae2db7bb-dd2f-4d26-8294-150f84c51559

 

Release of Smap HTML / iPhone Evaluation Tools

Leave a reply

Announcing Smap Evaluation Tools Server version 13.09.  New functionality includes:

  1. Support for completing surveys on a web browser. Browsers need to support HTML5 and can be on a laptop, iPhone, Android or Windows Mobile .
  2. Automatic dropping of duplicate surveys.  Previously the duplicate would have been marked as “bad” so that it did not appear in exported results or graphs however these duplicates could still be seen in table views.
  3. Calculate questions are now shown in the exported spreadsheet in their correct position relative to other questions in the survey.  So if you add a question to calculate the age in months after a date of birth question, then age in months will appear after DOB in the exported results.
  4. If you explicitly specify a “start time”, “end time” or IMEI question in your survey then these are no longer automatically added by Smap as well

WebForm Evaluation Tool

webform

Screen shot of web form in Chrome browser on a  Mac laptop

Features

  • Uses the open source Enketo forms engine
  • (Should) work in any HTML5 capable browser
  • GPS coordinates
  • Multi- choice and single select questions
  • Integer and text questions
  • Uploading of photographs (on some browsers, those with better HTML5 support)
  • Repeating groups
  • Relevance
  • Works offline and saves results for submission later when there is a network connection

Issues

The web forms feature is really a beta release so you can expect to find some issues.  Report any you find and we will try to get them fixed in the next release.  Known issues include:

  1. Some odk 1.4 functions that depend on javarosa do not work.  For example the new date time functions.
  2. Surveys that have images in their questions or options will not work with webForms
  3. Large surveys may be slow particularly when downloading or expanding a conditional group
  4. Safari on iPhone has some additional restrictions:
  • File uploads are not supported on Safari / iPhone
  • Expansion of conditional groups that are made relevant based on responses to prior questions do not work

Phone Browsers

iphone

Situation Assessment using Safari on an iPhone

android

Situation Assessment using Chrome on Android

 

 

 

 

 

 

 

Applications for the web forms feature

The ability to complete forms on web browsers adds a lot of flexibility to your data collection. For example it is now possible to send out an email or tweet with a link to a survey that the recipient can complete on their browser.

Installation

This is straightforward and very similar to previous installation.  Updated instructions are in the README file.  If you have an existing smap installation you should be able to just run the webformsInstall script to update the platform to support webForms (MySQL and PHP).  You can then deploy the updated application code as before.  Contact me if you have any issues.

If you are installing your own server you can get it from here.

Deployment onto Smap Managed Servers

I will progressively deploy this release onto the servers managed by Smap Consulting over the next few days.  However the ability to complete surveys using WebForms may not suit everyone.  Hence I will disable this functionality by default on each server unless you request that it be switched on.

Acknowledgements

Web Forms uses the excellent Enketo toolkit. This toolkit has been integrated into Smap by the following undergraduate students from RMIT who have done an excellent job.

  • HeeSien Ooi
  • Kar Ming Clement Kam
  • Kiman Sze
  • Mohamad Bahaa Maarouf
  • Shishir Chawla
  • Teck We Kee

 

Release of fieldTask v3.0

Leave a reply

Announcing fieldTask version 3.0. This release includes the changes in the odkCollect 1.4 release (thanks to the ODK team) along with the ability to send partially completed forms back to the server. You can download version 3 from google play and from the smap development server. It will be deployed onto other servers as they get upgraded.

ft3

fieldTask v3.0 Home Screen

1) Change Location. [Provided by ODK 1.4] There is a new ‘placement-map’ appearance for location questions.  If you use this “appearance” setting on a question of type “geopoint” then your current location will be shown on a map and you can do a long press on the map to change the position that will be recorded. The meaning of the 4 icons on the map is:

        1. Exit map without saving location
        2. Save location and exit map
        3. Place marker at current location
        4. Zoom to marker

location placement

2) New functions for date processing. [Provided by ODK1.4] Refer to the survey editing guideline and the example xls template.

3) Improved styling for phones with Android 3+. These will show an action bar with icons for some of the menu options. On smaller phones only one or two of the icons will be shown, the other menu options are accessible though the menu button.

icons

Android 2.2 phone menu showing all the icons. Android 3+ phones will show some icons on title bar, remaining menu options will be text when accessed through menu.

4) Access to question responses within repeating groups. [Provided by ODK 1.4].  This allows you to create one repeating group to get all the names of people in a household, then in a second repeating group you can ask detailed questions about each member of the household and refer to the name recorded for that householder. Refer to survey editing guideline and the example xls template.

5) Ability to defer constraint and “required” validation to the end of the survey. [Provided by ODK 1.4]. This can be enabled in the settings.

6) Sending partially completed forms back to the server.  If you have to save a survey half way through because it cannot be completed you can optionally send those results to the server.  After selecting “submit finished data” from the menu, press the menu button again and select “change view”. This will have two options “Show Complete Forms” and “Show Complete and Incomplete Forms”.  If you select the second option you will be able to see the saved incomplete forms and send them back to the server.

There will be an update to the server released by the 18th of October that will mark these incomplete surveys as “bad” with a reason of “incomplete”.  Hence they will not be included in exported spreadsheets or graphs unless the analyst marks them as “good”. If it is critical for you not to allow incomplete forms to be sent until the server is updated to mark them as bad by default, then defer use of fieldTask 3.0 until after the 18th of October.

Milestone 23,000 surveys in 100 days

2 Replies

Four years ago, in February 2009, a team of students from RMIT completed the first production survey for Smap. If I recall correctly around 100 households were surveyed in Cambodia.  The students did a great job getting the system working in particular with the Khmer script.  This survey was for World Vision and Tim Costello was kind enough to write a nice letter in thanks.

tim

 

Since then IBM has dropped out of the project, or to put it another way, I left and started Smap Consulting.  RMIT and World Vision are still very much involved.  And now, just over 4 years later, I think we have hit something of a milestone 23,000 surveys were submitted onto Smap hosted servers in the last 100 days.

It seems the technology may have reached a tipping point where it is much cheaper, easier and more effective to conduct assessments with mobile phones instead of paper.