Statistical Analysis of Public Tweets to Provide Early Notification of Emergency Events

CSIRO have been developing software for monitoring social media.  Last monday I went to Canberra to talk to Ron Jones who is responsible for business development of these products.  We met, near his office on the ANU campus, at the Purple Pickle cafe.

We discussed two products but I am going to focus on the Emergency Situation Awareness application (ESA) for this blog post since its very relevant to the work that Smap Consulting does. The title of this post is a quote from a video produced by CSIRO showing how ESA works.

My understanding is that ESA analyses the frequency of occurrence of key words or hashtags in tweets over a period of several months. Then if there is a deviation from the background level of a word during a 2 minute window an alert will be created at one of 4 levels depending on the amount of deviation; green-blue-purple-red.  Here is a screen shot from the video showing alerts for the word storm around about the time a tropical storm hit Brisbane.



The second column is the date and time that the alerts began, the third column the duration that an elevated alert level was maintained and the 4th column the number of alerts during that period. ESA has an API that allows an external system to interrogate the statistical server.

This would be a great tool to integrate with Smap and Ushahidi to get a better picture of the situation after a rapid onset emergency.  ESA could be used to identify new events while Smap can be used to provide an assessment of that event.  The results can then be shown on a tool such as the Speed portal, further filtering the “firehose” of information that emergency workers can be subjected to.

Getting this tool applied to rapid onset emergencies in developing countries would require a little bit of effort and cash. For example currently only tweets in Australia are monitored.  Possibly we would need a server for Africa or Asia etc. However it looks very promising to me.


2 thoughts on “Statistical Analysis of Public Tweets to Provide Early Notification of Emergency Events

  1. Viv McWaters

    Hi Neil
    This is a great development and addition to Smap. You mentioned it’s use for rapid on-set emergencies. I wonder if there is also a way to monitor chronic emergencies? By their very nature they are harder to determine when a tipping point has been reached. (LOL – I originally had tipping pint written. Something that could monitor when someone had tipped too many pints would also be good. I’m sure the quality of their tweets would deteriorate with additional pints 🙂

    1. Neil Penman Post author

      Hi Viv,
      There would need to be some adjustments to monitor chronic slowly moving emergencies as the CSIRO technology is set up to detect rapid changes in message rates over periods of a few minutes. Establishing longer term trends would presumably require quite different statical analysis. I agree it would be really useful. I worked on an evaluation of the response to the 2011 drought in the horn of Africa. Although one of the findings was that the drought was in fact timely it was controversial. Its very difficult to pick that tipping point. Could be useful to have some statistics that can give a measure to to this.

      Perhaps combining neural networks with the Emergency Situation Awareness application could be be worth trying. Neural networks are great at picking out patterns in noisy data and could be trained by feeding information from previous chronic emergencies since after the fact its much easier to pinpoint when the disaster should have been declared.

      Unfortunately EAS has not been added to the Smap tool set yet. I’m looking for partners in organisations that want to try it out in emergency response efforts. I reckon there would be plenty of potential health or recreational uses for a tipping pint application too. There are 10 government agencies that have put money into developing ESA further I wonder if the health department is one!



Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.