Author Archives: Neil Penman

Smap version 14.02 Multi Tenant

Leave a reply

Announcing the release of Smap Server version 14.02.  The major new feature is that multiple organisations can now be put on a single installation of Smap, so called multi-tenant.  The administrators or analysts in each of these “organisations” cannot see projects or surveys in any other organisation however within each organisation the administrator can still create projects and users.

Multi-Tenant

When you install the Smap server it will create a user with a login id of “admin”.  This user has the “organisational administration” role.

multi1

If the user has the organisational admin role, they will be able to add organisations and move users and projects to other organisations.

multi2

  1. Select “Add Organisation” to create a new organisation
  2. Select users and projects then click “Move” to move these projects and users to a different organisation.  The Organisation Administrator can move themselves as well as other users.

Users, including administrators, without the organisational administration role will only be able to work within the organisation that they have been assigned.  Each user can be assigned to only one organisation.

The name of the organisation is shown after the users name at the top right of each screen.  An ‘@’ symbol separates the organisation name from the person’s name.

multi3

Export surveys with location as KMZ or VRT

Two new export formats have been added in this release.  Due to the increasing number of format options the radio buttons that were used to select the format have been replaced by a drop down selection.

  1. KMZ / KML files can be loaded directly into Google Earth.
  2. VRT files are accompanied by a CSV file holding the data.  This is useful for geospatial processing including being imported into a GIS such as google maps engine. VRT is a document format used for describing vector or raster geospatial data. In this case the actual data is in a CSV file and the VRT file identifies the type of geometry (Linestring, Polygon or Point).

    Smap can also export shape files however they are limited to 255 attributes and even worse the length of the attribute name is restricted to 10 characters which results in attribute names being truncated and difficult to differentiate. Whereas the overall limit on the number of attributes per feature in google maps engine is 550 with a maximum name length of 64 characters.

Check on maximum questions per form

There is a limit of 1,600 columns in each database table.  Because the results from each form are put into separate database tables then this imposes a limit on the size of each form. Each question is stored in its own column and also each option in a multiple choice question gets its own column.  10 columns have also been reserved for the unique key, key to the parent table, start date, end date, device id and user identifier.

When uploading a template the server now checks the number of questions and multiple choice options and will report an error if there are too many.

msg

Example error message when there are too many questions in a form

Fortunately it is straightforward to move excess question into another form and this should not affect either data collection or analysis. More details here.

Allow access to attachments without an account

The default installation no longer requires a user to have a valid user id and password to view an attachment such as a picture.  Instead each attachment has a name created from 28 random characters.  For example: 4e2252ad-39db-46cb-b50d-aubb94ca353a.PNG. This will simplify integration with other systems while still preventing unauthorised access to attachments.

Some Bug Fixes

  1. Template files with a suffix of “.XLS”, that is in capitals were not being identified as Excel files. Now the test is case insensitive and file extensions can include capitals.
  2. Results from surveys with unicode characters, that is Arabic, Chinese, Japanese, Thai etc, could not be be exported.  These can now be exported and the name of the download file will be the same as the survey name.

Controlling access to survey results

Leave a reply

There are 3 roles that can be given to users od Smap Server:

  1. Admin.  Can create new projects and assign themselves, or any other user, to projects
  2. Analyst.  Can view and export results, create graphs, load new survey templates.
  3. Enum. Can download survey templates to a phones and submit results.

Hence you can restrict a user from accessing surveys that they are not involved with, by putting surveys into projects.  Users can then be assigned only to those projects that they are allowed to access.  As long as the user does not have the admin role then they will not be able to give themselves access to other projects or create new projects.

 

An issue with the current solution is that you may have multiple areas of your organisation that work independently hence you will want to create a user with the “Admin” role for each area.  But these admin users will be able to view and delete survey data collected by other areas.

The next release of Smap Server (14.02) due in late February will address this issue by adding the concept of an “organisation”. Even admin users, within 1 organisation will have no access to data in other organisations.  You will be able to create an organisation for each area of your company.

Smap Version 14.01 Task Management

Leave a reply

Announcing the release of Smap Evaluation Tools Server version 14.01.  This release includes

  • Task management: the ability to filter the surveys that are used as locations for new tasks
  • Task management: Printing of task allocations across multiple pages
  • Support for Internet Explorer 10+ as the browser
  • Improved reporting of errors when uploading templates

Task Management

Task Filtering

When you create new tasks from an existing survey you can now specify a filter on one of the questions in the existing survey.  Only survey locations that match that filter will be added to the new task group. Select the checkbox called “Filter tasks” to enable filtering.

tm1

Task Filtering

 

In the following example the previous survey recorded the paths of various cable types including copper and fibre.  However for the new task I only wanted the surveyor to go to the cables of type copper so I enabled filtering and selected the option of “copper”

tm2

Tasks filtered by cable type of copper

Printing Task Sheets

When you select the “Print” button to print a task sheet the tasks are now shown on more than one page.  This attached PDF shows this. Note the address parameters that have been added to help locate the location of each task.  In this case these include the age, ethnicity and a picture.

 

 

 

Setting the default language

Leave a reply

One of the new features in version 13.12 of the Smap Server is the ability to set the default language.  Previously this was always the last language entered in the survey work sheet.

There are two ways of setting the default.

1. The xlsform standard way is to add a new worksheet “settings”.  Provide a single column with a heading of “default_language” enter the default language under that column.

lang1

Default Language in Xlsform

2. The other approach is to specify the default language directly in the smap server.  You can change the default here any time without having to reload the template.  You do this from the template management screen. Select “view/edit” next to the name of the survey you want to change.

lang2

Default language on Smap Server

Then select the default language from the drop down list.

 

 

 

 

Smap Version 13.12 available for download (and new download Site)

Leave a reply

You can now find a list of Smap Server versions available for download at http://www.smap.com.au/downloads.shtml. Version13.12 as well as training documents are now available from that location.

Older versions of the server can still be downloaded from the Google code site.  https://code.google.com/p/smap-suite/downloads/list. Search for “Deprecated Downloads”.

The download is actually hosted on Amazon S3.  This move was required because Google no longer allow new downloadable files to be added to https://code.google.com.  Hence the training documents have all been moved to google drive and the large server files to Amazon S3.

Creating Time Series Graphs

Leave a reply

With the release of version 13.12 of Smap Server you can create graphs, maps and tables that show data against a time axis. This post describes a series of steps that proceed from a very simple bar chart to a, still simple, time series graph.  You can of course create these time series graphs in one step however you might use this approach of progressively refining your settings in order to show progressively increasing details.

The data in the examples is artificial and is from a survey that collects water depth in a well from two different  locations.  The water depth question was of type “integer”, the location question was of type “select_one”.

1. Bar chart of water depth

First select “Add Panel” to create a new panel, select your survey, and your question then press the “save” button. You will see something like the following. This shows the average water height for all locations at all times.

g1

Bar Chart

2. Bar chart of Water Depth by Location

Then specify the location as the “group by” question. This will show the average water height at each location for the entire time of the survey.

g2

Bar chart with “group by”

3. Time Series Plot of Water Depth by Location and Day

Then select the date question to use as the x-axis. This settings field is just under the name of the survey. Also select a period for the time axis. In this example I selected “day”.  The settings panel will then look something like this:

s1

Settings Dialog Configured for Time Series

The resultant graph shows average water height per day and per location. Note the gap in data between the 14th of January and the 5th of February.  No measurements were taken during this period.

g3

Time series chart by location by day

 4. Time Series Plot of Water Depth by Location and Week

If you then open the settings and change the period to “week” the x-axis of the graph will be updated so that average water height per week and per location is shown.

g4

Time series chart by location by week

 

Smap Server version 13.12 with Time Series Analysis

Leave a reply

Announcing the release of Smap Evaluation Tools Server version 13.12. The time series analysis functionality in this release has been funded by the Speed Evidence project. This is to assist in the long term response to Typhoon Haiyan.

New functionality

  1. Time Series Analysis.  Showing trends over time using graphs, maps and tables in the analysis module.
  2. Server side calculations.  There are only two functions available at the moment; area of a polygon and length of a line or the perimeter of a polygon.  However other functions could be added such as the calculation of Z-score anthropometric indicators.
  3. Export of attributes from parent forms to shape files.  If you select a child form containing a geometry and export to a shape file then all the attributes of the parent (up to the maximum of 255) will also be exported.
  4. Specification of default language.  The default language to show on the phone can now be specified by selecting “view/edit” link next to the survey name on the template management page.
  5. Support for Internet Explorer has been improved however it is still not supported, that is I haven’t tested this release against IE.

Availability

Hosted servers will be progressively updated over the next week or so.  The download for this release will be available by the 19th of January.

 

 

Time Zones in Reports and Graphs

Leave a reply

Its timely to look at the use of time zones in reports and graphs since we will shortly be releasing a version of Smap that includes trend analysis.  This will allow you to create graphs and maps  showing trends in data against time.

The most common date columns in Smap reports are the start and end of the survey.  When these are shown in a table on the Smap server the time zone is included. For example the following is from a survey that started at 7:10pm in a time zone GMT + 11.

  • 2012-03-13 19:10:27.412+11

Note the time zone is shown as +11. When the survey is exported to Excel the time zone information is removed and the time is shown as GMT.

  • 13/03/2012  8:10:27 AM

This is because Excel can’t include the time zone in a date cell.  (At least I don’t think it can). However you can modify the time back to local time.  Dates in Excel are stored as floating point numbers.  If you add “1” to this date it will increment by 1 day.  So to change the time to a time zone of GMT + 11 you can add 11/24.

In a future release of Smap I will add an extra column for the time zone of the survey so that this information is available in the report.

Now getting back to the trend analysis. The times in the x-axis of trend graphs will be shown in the local time of the survey without timezone information.  So for the example above where the time interval is set to show results by the hour.  That survey result will fall within the x-axis period of:

  • 2012-03-13 19

In summary all dates and times in reports are in local time except when exported to Excel when they are shown as GMT.

Release of Smap Data Cleansing

3 Replies

Announcing the release of Smap Evaluation Tools Server version 13.11. This version has concentrated on some data cleansing functionality which as been requested by a team assessing the needs of people affected by Typhoon Haiyan.  The Speed Evidence project has provided funding.

New functionality

  1. Updating data on the server prior to export. (Data Cleansing).
  2. Ability to hide the graph legend as this sometimes obscures the data. http://blog.zarkman.eu/hiding-gaph-labels/
  3. Bundling of Smap Uploader
  4. Support for Apache 2.4

Data Cleansing

The following video shows how to update text values on the server.

The analyst training material has also been updated to included more detailed instructions data cleansing.

Other changes

A bug that prevented the creation of graphs when question names included dashes has been fixed.

Uploading of Polygons and Lines has been made more robust.  Previously if a valid Polygon had not been recorded (more than 2 points) then the survey was not uploaded.  Now the rest of the data is uploaded and the invalid Polygon / Line is discarded.

Availability

A downloadable tar file of the updated server should be available tomorrow.  Hosted servers will be progressively updated over the next few days.

Limits on the number of questions per form

1 Reply

Recently a Smap customer, who are conducting a hospital audit, encountered problems loading a survey that had just under 2,000 questions.  The problem was caused by a limit in the Postgres database. There can be no more than 1,600 columns per table.  In a survey you can reach this limit either by specifying:

  • 1,600 questions
  • or 160 multiple choice questions with 10 options in each, since a separate column is used for each option in a multiple choice question
  • or a combination of the above

The solution to the problem is to enclose some questions within a repeat group with a “repeat_count” of “1”.  These questions will be written to a separate table in the database.

While the data collector is filling in the survey they will not be aware of this repeat group as it is set to automatically repeat just once.

When you export the survey results, if you select the “pivot” option, then the records from this “begin repeat” subform will be combined with the records from the main form.  However you will also have the option of exporting the main form and its sub form(s) separately.  I’ve seen exports of surveys to spreadsheets with over 3,000 columns and I’m not aware of any constraints on this.

I was able to load the large hospital audit survey after adding this begin repeat, end repeat solution. A future release of Smap will check that the maximum number of questions per form is not exceeded. If it is an error message will be shown.