Announcing the release of Smap Server version 14.02.  The major new feature is that multiple organisations can now be put on a single installation of Smap, so called multi-tenant.  The administrators or analysts in each of these “organisations” cannot see projects or surveys in any other organisation however within each organisation the administrator can still create projects and users.

Multi-Tenant

When you install the Smap server it will create a user with a login id of “admin”.  This user has the “organisational administration” role.

If the user has the organisational admin role, they will be able to add organisations and move users and projects to other organisations.

1. Select “Add Organisation” to create a new organisation
2. Select users and projects then click “Move” to move these projects and users to a different organisation.  The Organisation Administrator can move themselves as well as other users.

Users, including administrators, without the organisational administration role will only be able to work within the organisation that they have been assigned.  Each user can be assigned to only one organisation.

The name of the organisation is shown after the users name at the top right of each screen.  An ‘@’ symbol separates the organisation name from the person’s name.

Export surveys with location as KMZ or VRT

Two new export formats have been added in this release.  Due to the increasing number of format options the radio buttons that were used to select the format have been replaced by a drop down selection.

1. KMZ / KML files can be loaded directly into Google Earth.
2. VRT files are accompanied by a CSV file holding the data.  This is useful for geospatial processing including being imported into a GIS such as google maps engine. VRT is a document format used for describing vector or raster geospatial data. In this case the actual data is in a CSV file and the VRT file identifies the type of geometry (Linestring, Polygon or Point).

   Smap can also export shape files however they are limited to 255 attributes and even worse the length of the attribute name is restricted to 10 characters which results in attribute names being truncated and difficult to differentiate. Whereas the overall limit on the number of attributes per feature in google maps engine is 550 with a maximum name length of 64 characters.

Check on maximum questions per form

There is a limit of 1,600 columns in each database table.  Because the results from each form are put into separate database tables then this imposes a limit on the size of each form. Each question is stored in its own column and also each option in a multiple choice question gets its own column.  10 columns have also been reserved for the unique key, key to the parent table, start date, end date, device id and user identifier.

When uploading a template the server now checks the number of questions and multiple choice options and will report an error if there are too many.

Example error message when there are too many questions in a form

Fortunately it is straightforward to move excess question into another form and this should not affect either data collection or analysis. More details here.

Allow access to attachments without an account

The default installation no longer requires a user to have a valid user id and password to view an attachment such as a picture.  Instead each attachment has a name created from 28 random characters.  For example: 4e2252ad-39db-46cb-b50d-aubb94ca353a.PNG. This will simplify integration with other systems while still preventing unauthorised access to attachments.

Some Bug Fixes

1. Template files with a suffix of “.XLS”, that is in capitals were not being identified as Excel files. Now the test is case insensitive and file extensions can include capitals.
2. Results from surveys with unicode characters, that is Arabic, Chinese, Japanese, Thai etc, could not be be exported.  These can now be exported and the name of the download file will be the same as the survey name.

There are 3 roles that can be given to users od Smap Server:

1. Admin.  Can create new projects and assign themselves, or any other user, to projects
2. Analyst.  Can view and export results, create graphs, load new survey templates.
3. Enum. Can download survey templates to a phones and submit results.

Hence you can restrict a user from accessing surveys that they are not involved with, by putting surveys into projects.  Users can then be assigned only to those projects that they are allowed to access.  As long as the user does not have the admin role then they will not be able to give themselves access to other projects or create new projects.

You can now find a list of Smap Server versions available for download at http://www.smap.com.au/downloads.shtml. Version13.12 as well as training documents are now available from that location.

Older versions of the server can still be downloaded from the Google code site.  https://code.google.com/p/smap-suite/downloads/list. Search for “Deprecated Downloads”.

The download is actually hosted on Amazon S3.  This move was required because Google no longer allow new downloadable files to be added to https://code.google.com.  Hence the training documents have all been moved to google drive and the large server files to Amazon S3.

Its timely to look at the use of time zones in reports and graphs since we will shortly be releasing a version of Smap that includes trend analysis.  This will allow you to create graphs and maps  showing trends in data against time.

The most common date columns in Smap reports are the start and end of the survey.  When these are shown in a table on the Smap server the time zone is included. For example the following is from a survey that started at 7:10pm in a time zone GMT + 11.

• 2012-03-13 19:10:27.412+11

Note the time zone is shown as +11. When the survey is exported to Excel the time zone information is removed and the time is shown as GMT.

• 13/03/2012  8:10:27 AM

This is because Excel can’t include the time zone in a date cell.  (At least I don’t think it can). However you can modify the time back to local time.  Dates in Excel are stored as floating point numbers.  If you add “1” to this date it will increment by 1 day.  So to change the time to a time zone of GMT + 11 you can add 11/24.

In a future release of Smap I will add an extra column for the time zone of the survey so that this information is available in the report.

Now getting back to the trend analysis. The times in the x-axis of trend graphs will be shown in the local time of the survey without timezone information.  So for the example above where the time interval is set to show results by the hour.  That survey result will fall within the x-axis period of:

• 2012-03-13 19

In summary all dates and times in reports are in local time except when exported to Excel when they are shown as GMT.

Announcing the release of Smap Evaluation Tools Server version 13.11. This version has concentrated on some data cleansing functionality which as been requested by a team assessing the needs of people affected by Typhoon Haiyan.  The Speed Evidence project has provided funding.

New functionality

1. Updating data on the server prior to export. (Data Cleansing).
2. Ability to hide the graph legend as this sometimes obscures the data. http://blog.zarkman.eu/hiding-gaph-labels/
3. Bundling of Smap Uploader
4. Support for Apache 2.4

Data Cleansing

The following video shows how to update text values on the server.

The analyst training material has also been updated to included more detailed instructions data cleansing.

Other changes

A bug that prevented the creation of graphs when question names included dashes has been fixed.

Uploading of Polygons and Lines has been made more robust.  Previously if a valid Polygon had not been recorded (more than 2 points) then the survey was not uploaded.  Now the rest of the data is uploaded and the invalid Polygon / Line is discarded.

Availability

A downloadable tar file of the updated server should be available tomorrow.  Hosted servers will be progressively updated over the next few days.

I have released a patch to the Smap Analysis software in response to a request from the team in the Philippines who are responding to Typhoon Hainan.  This allows you to hide labels on a graph by clicking on an icon labelled “L” at the top right of the panel.  Clicking on the “L” again will show the labels.  This is available in the dashboard and in reports.

Graph with hidden labels. Note the “L” icon at top right.

When the labels are hidden you can see what each of the bars represent by moving your mouse over the bar.  This will show the label just for that bar along with the bar’s value.

This patch to the server also fixes a bug so that the description added by the analyst is now shown on the report.

This release is only available on a limited number of the hosted servers at the moment but will be included in the next monthly release.

The Smap Uploader tool is now available.  This will be bundled with the server in the next release of Smap.

Ideally you should never need to use Smap Uploader!  It is much more straightforward to connect the phones directly to the server over a network.  However you may have backed up your surveys and then the phone has been damaged, or the uploads from that phone may not work for some reason. In these cases Smap Uploader provides a fall back.

Click on the Smap Uploader button on http://dev.smap.com.au to download. You will need Java 6 or later https://java.com/en/download/index.jsp.  Save the downloaded file somewhere where you can find it again, possibly on your desktop.  Double click to start it.

Smap Uploader Screen

You can copy the forms from your phone using a USB cable.  The exact process may vary depending on the phone but commonly you will need to attach as a USB mass storage device. The forms can be founds under /sdcard/fieldTask/instances.  Each one will have its own directory where attachments are also stored.

In Smap Uploader choose the above folder and fill in the server name, your user id and your password. Then press submit. Progress will be shown in the status panel along with an hourglass.

If you have to stop sending part way through you can just exit Smap Uploader.  It will resume from the point you left off next time you select the form folder and press submit.

You may end up sending forms multiple times.  For example some forms could have been sent from the phone and then also sent from Smap Uploader.  In this case the duplicate forms will be discarded by the server.

The Speed Evidence project has released a video showing the features of the Information Management Portal. This is looking really good.

However up until today media files such as pictures that were included in a table of data or a map were not being copied across into the report shown by the portal. The Speed Evidence portal pulls assessment reports from the Smap Server using the reports API. These reports consist of a graph, map, table, picture, video or audio file that have been collected using the Smap Mobile Phone tools.   The report contents have to be copied into a report location on the server so that they can be accessed using the report identifier rather than a user name and password.  This report identifier will look something like this “293ffa7b-70j7-4dc0-gb4a-57fcb45a20a3” and uniquely identifies the report.

This issue has now been fixed. When a report is created now all the media files that are included in tables and maps are copied over to the reports area and assigned the report identifier. You can see this in the example report below which is a table of data containing pictures grouped by the nationality of the respondent.

https://dev.smap.com.au/surveyKPI/reports/view/ae2db7bb-dd2f-4d26-8294-150f84c51559

Four years ago, in February 2009, a team of students from RMIT completed the first production survey for Smap. If I recall correctly around 100 households were surveyed in Cambodia.  The students did a great job getting the system working in particular with the Khmer script.  This survey was for World Vision and Tim Costello was kind enough to write a nice letter in thanks.

Since then IBM has dropped out of the project, or to put it another way, I left and started Smap Consulting.  RMIT and World Vision are still very much involved.  And now, just over 4 years later, I think we have hit something of a milestone 23,000 surveys were submitted onto Smap hosted servers in the last 100 days.

It seems the technology may have reached a tipping point where it is much cheaper, easier and more effective to conduct assessments with mobile phones instead of paper.