Tag Archives: Smap Consulting

Version 14.09 – Form Refresh and Export to Neo4j

Leave a reply

Version 14.09 of Smap is available for download.

Refresh – Synchronising phones with the server

The refresh button on fieldTask now does more than just clear the submitted results from the screen. It also now refreshes the forms that you have on your phone and submits any finalised results.  Note you will need to download fieldTask version 2.3.5 from your server on to your phone to use this new feature.

Pressing refresh will now:

        • Download new forms from the server that have been assigned to the user of the phone
        • Show available forms with a yellow clipboard icon
        • Update forms on the phone that have a new version
        • Delete forms from the phone for which the user no longer has access
        • Submit finalised results from the phone to the server
        • Notify the server of the list of forms and their version that the user has downloaded to their phone

You can select a form from the task list screen to complete a survey.  The name, version and project containing the form are shown next to the icon.

device-2014-09-20-090412

Phone with two forms

This new feature should simplify the training of the data collectors as the steps they need to download new forms, start a survey and submit results are all simpler.

Checking that data collectors have the correct version of a form

Sometimes it can happen that data collectors go out into the field with the wrong version of a form on  their phone.  At worst this may have an impact on quality of the collected data and at best you will need to do some messy combining of the data from the two forms prior to analysis. This new feature makes it much easier to check that all the data collectors have the correct version of the form.

From the monitor page select forms.  This will show the users, phones, downloaded forms and the versions of those forms that have been downloaded.  Any users / phones that have not downloaded a form or have an older version are highlighted.

monitor

Monitor Downloaded Forms

 Exporting results to Neo4J / Thingsat

You can now export results to the Neo4J graph database. This change was developed in collaboration with thingsat.  The following video shows how this can be done.

Other Changes

  1. You can edit collected data using web forms.  This capability is enabled by an administrator for your organisation.  By default it is off.  If you select a table showing data for the whole of survey then there will be a button next to each data record that will open the survey results in a web form. If the data is updated the old results will be marked as bad, and then the updated results added.
  2. When you attempt to submit results from a phone for a survey that has been deleted on the server then the error message will be “Not Found” rather than “Not Authorised”
  3. The error message shown, if you upload a survey template that has duplicate options, has been made clearer

Smap Version 14.03 Forwarding

3 Replies

Smap version 14.03 has finally been released.  It should have been available from the end of March. The next release should be a small one and hopefully we can get back to the end of month regular release date.

The major new feature is forwarding of surveys.  Other changes are improvements to monitoring, support for larger result sets (paging) and a more laptop friendly deployment.

Monitoring

Totals

When you select “Monitoring” by default you will be shown total counts for success, errors, duplicates and “not uploaded”.

monitor

Totals are only available in Table view.  In order to view locations of uploaded data you need to first select the source as “Instances” then select “show as map”.

Filtering

You can filter the uploads (or forwards) that you view in the monitor page by their status. Success, errors, duplicates and “not uploaded”.  Hence you can uncheck all statuses except for “Error” in order to see which uploads in your project or survey were reported as being in error.  When showing instances rather than totals the last 200 records are still shown.  However if you apply a filter to only show errors then it will be the last 200 errors that are shown.

Paging

Paging is now available in the Monitoring screen and for the survey table view in the dashboard.  200 records are shown by default but you can then select a forward / reverse button to move forward and back through the results. These buttons are located at the top of the table and are labelled “>>>” for forward and “<<<” for back.  They will be disabled unless there are additional pages available to view.

Total Result Records

In the dashboard, when you view all survey results as a table, then the total count of surveys will be shown as well as the count of “bad” records. This wil be a faster way of getting a count of all surveys loaded than having to export to a spreadsheet.

Deployment

Changing URL of the server

Previously it was assumed that the address of the server would not change.  This allowed the server to get away with storing the full url of all survey attachments in the database.  However this assumption is completely untenable if Smap Server is deployed on laptops.  The server URL can now be changed and all the attachments will still be accessible.

Simplified deployment

Almost all the editing of files that had to be done during the installation process is now not needed.  The one exception is that the IP address of the server still has to be set in the install script.   This is only required for webForms and will hopefully be removed soon.  So, assuming you are installing on a clean Ubuntu machine, the new installation process is:

  1. Download and extract the release
  2. change directory to smap/install
  3. Edit the install script to add the IP address (temporary)
  4. Type ./install.sh

This will also now run the deploy script so once it finishes you shoulld be able to log on and use the server.

This release will be added to the downloads page over the coming weekend.  There will be another blog post to announce its availability.

Bug Fixes

  1. Error in retrieving languages on the audit page fixed.  This bug prevented auditing.
  2. If the first record in survey had an invalid geometry then export to shape files would fail. An example of an invalid geometry is recording only one point when a polygon or line was requested.

 

Forward Results to another Server

Leave a reply

Beginning in Smap version 14.03 you can add forwarding to a survey template.  This will result in each submitted result being forwarded on to the target server.

Key characteristics

  • Each forward link sends results from a single survey template to a single remote server
  • You can add multiple forward links to a template allowing you to forward results to many remote servers
  • Submissions are sent in one direction only. Hence if the target server is receiving results from multiple sources it will have a different view of the results than each of the sources
  • Forwarding links can be added to the target server as well. These can point back to the original source server but it ahs to be to a different survey on that server
  • Existing results will be forwarded if you add forwarding to a template that already has submitted results
  • Results will queue until there is a network connection before being forwarded
  • The target server can be another Smap server or any server that supports Java Rosa such as ODK Aggregate

Offline laptops

One way of using this forwarding feature is to install Smap on a laptop.  This laptop can be used to aggregate and analyse data collected on mobile phones in areas that do not have a reliable Internet connection.  You can upload the results from the phones to the local laptop over a wi-fi hotspot.  Once you connect the laptop to the Internet it can forward the results onto a cloud server so that they can be worked on by a distributed team.

Community Laptops

image (6)

Community laptops are an extension of the “offline laptop” concept.  Each community can maintain its local view of data collected on the phones.  However they can set up a forward link so that a consolidated view of all the data from all the communities can be aggregated.

Demonstration

The following video shows the forwarding of results from a Smap Server to an ODK Aggreate server.

How to add a forward link

  1. Create the same survey template on the local and remote server.  These templates can have different names on each server but should be the same template.  A good way to do this is to download the template as an XML file from one of the servers and then upload it to the other.
  2. On the source server, find your template on the template management screen and click the upward pointing arrow next to its name
  3. Click the button “Add Forward”
  4. Enter the url of the destination server
  5. Enter a user ID and password.  You should use a user id that only has privileges to complete surveys (enum) as the password is stored in the database. So don’t use your Admin user id!
  6. Press the button to refresh the list of available surveys on the remote server
  7. Select the survey that you want to update
  8. Save

You can monitor forwarded results on the monitor page by selecting the “forward” radio button.

Smap Server version 13.12 with Time Series Analysis

Leave a reply

Announcing the release of Smap Evaluation Tools Server version 13.12. The time series analysis functionality in this release has been funded by the Speed Evidence project. This is to assist in the long term response to Typhoon Haiyan.

New functionality

  1. Time Series Analysis.  Showing trends over time using graphs, maps and tables in the analysis module.
  2. Server side calculations.  There are only two functions available at the moment; area of a polygon and length of a line or the perimeter of a polygon.  However other functions could be added such as the calculation of Z-score anthropometric indicators.
  3. Export of attributes from parent forms to shape files.  If you select a child form containing a geometry and export to a shape file then all the attributes of the parent (up to the maximum of 255) will also be exported.
  4. Specification of default language.  The default language to show on the phone can now be specified by selecting “view/edit” link next to the survey name on the template management page.
  5. Support for Internet Explorer has been improved however it is still not supported, that is I haven’t tested this release against IE.

Availability

Hosted servers will be progressively updated over the next week or so.  The download for this release will be available by the 19th of January.

 

 

Connected Arabic fonts in Android 2.3

3 Replies

I’m working with a customer that is conducting a survey in Iraq using Samsung Galaxy Y Duos (android 2.3) and fieldTask. Later model phones (android 3+) worked well with Arabic but not these earlier versions. Even after rooting the phones and installing new fonts it still didn’t connect the characters.  See below a screen shot using Dr Ben Ox Arabic fonts which seemed to give the best results.
arabic2 (1)

Fortunately there is an Arabic reshaping library available from https://github.com/agawish/Better-Arabic-Reshaper/ which can be used with these old phones. I added ArabicReshaper.java and ArabicUtilities.java from this library to the odkCollect library that performs survey completion in fieldTask. I then modified the “QuestionWidget.java”, “SelectOneWidget.java” and “SelectMultipleWidget.java” files in odkCollect to call the “reshape” function when setting the text. See line 12 below.

[code language=”java” highlight=”11,12″]
protected void addQuestionText(FormEntryPrompt p) {
String imageURI = p.getImageText();
String audioURI = p.getAudioText();
String videoURI = p.getSpecialFormQuestionText(“video”);

// shown when image is clicked
String bigImageURI = p.getSpecialFormQuestionText(“big-image”);

// Add the text view. Textview always exists, regardless of whether there’s text.
mQuestionText = new TextView(getContext());
// mQuestionText.setText(p.getLongText()); smap
mQuestionText.setText(ArabicUtilities.reshape(p.getLongText())); // smap
mQuestionText.setTextSize(TypedValue.COMPLEX_UNIT_DIP, mQuestionFontsize);
mQuestionText.setTypeface(null, Typeface.BOLD);
mQuestionText.setPadding(0, 0, 0, 7);
mQuestionText.setId(QuestionWidget.newUniqueId()); // assign random id

// Wrap to the size of the parent view
mQuestionText.setHorizontallyScrolling(false);

if (p.getLongText() == null) {
mQuestionText.setVisibility(GONE);
}

// Create the layout for audio, image, text
mediaLayout = new MediaLayout(getContext());
mediaLayout.setAVT(p.getIndex(), “”, mQuestionText, audioURI, imageURI, videoURI, bigImageURI);

addView(mediaLayout, mLayout);
}
[/code]

This improved the display significantly:
SC20130708-172712

This Arabic version of fieldTask is available for download from http://dev/smap.com.au. There are other widgets that could have this functionality added and presumably the arabic text typed by an enumerator in a text field should also be reshaped. These additional changes will be made on an as needs basis. The preferred option is to get a phone that supports Arabic fonts natively which would be a better option than having to root the phones to install new fonts.

Announcement of Smap Release – April 2013

2 Replies

Dashboard panels are now only viewable by the user that created them.  No more sharing!  The concept of sharing all dashboard panels worked reasonably well when a project team was working collaboratively on a project.  But there were still issues, for example when a team member wanted to do some experimentation with graphs.  However there were significant problems during workshops when 30 odd people were creating and deleting panels independently and the system was trying to keep all of the panels in sync.   So I came to the conclusion that no-sharing is better than sharing everything.   If anyone has any ideas on how to introduce some managed sharing back in to the dashboard panels let me know.

You can now specify address fields from the previous survey in follow up tasks.  See picture below:

tasks

fieldTask works a little differently.  The menu option “get new forms” now goes direct to the server.  If you want to delete forms or data from the phone you will need to select “delete forms” from the main menu.  

Other changes:

  • More informative error messages when uploading templates
  • Surveys can be set in settings to automatically upload when they have been completed (with odk1.3)
  • In settings you can specify the use of forward and next buttons instead of swiping (with odk1.3)
  • Vertices of polygon questions are now shown on map panels.  Apart from allowing the user to look at question responses on each point of the polygon the display of these points make it easier to find small polygons on the map.  (This one for Mark)
  • The usual assortment of bug fixes

Announcement of Smap Release – February 2013

Leave a reply

 

 

The february release of Smap is out of development and undergoing pre-release testing.

New Features:

  1. Multi-projects.  A user with administrator privilege can now create a new project and assign users to that project.  This allows you to manage separation of projects on the same server.  A user will only see the survey templates and results for the projects that they are assigned.
  2. Pictures, videos and audio can be included as survey questions or options.  Any or all of these media types can be added, along with text, to help an enumerator in completing a survey.  This capability has been on the phone for a while as it is part of odkCollect.  However the ability to manage this media on the server is new.  (video)
  3. Publish reports to a Facebook group.  This extends the current ability to publish a report via email.  A menu option named “Discuss” has also been added to allow you to link to a new Facebook app from Smap Consulting, that summarises the discussions on the reports as a word cloud and shows the locations of reports on a map.  (video)
  4. Added ability to specify an export of data as XLS or CSV.  For XLS exports the following formatting is now applied:

  • Dates.  Formatted correctly for excel and always set to GMT.  To change the timezone in excel add the hours difference divided by 24.  You can now do calculations such as subtracting _start from _end to get the duration of the survey.
  • _device formatted as number so it does not show as scientific notation by default.
  • Links added to locations.  If you click on the link it will show on a map where the survey was completed.
  • Links added to pictures, video and audio.

Bugs Fixed

  1. Hourglasses were not always being shown when the system was busy
  2. Name of survey in monitor screen was not being shown.
  3. Could not filter by survey name in monitor screen.Burn Down Chart
  4. Running out of database connections under heavy load

This diagram is of the project burn down and is taken from Acunote, which is a great tool for managing Agile projects. The chart shows the rate of progress across the month and confirms that development was completed on time.  Of course any tasks that could not be done prior to the release date, and that could be moved, were simply put into the sprint for the March release.  Completing “on time” for an agile project is usually a given!