Social marketers keep a close eye on the topics that are currently “trending” on Twitter. But what if you could predict those hot topics with 95% accuracy, 90 minutes before they hit the trend list?
MIT Associate Professor Devavrat Shah and his student Stanislav Nikolov say they have developed a proprietary algorithm which can in fact predict what topics Twitter’s own proprietary trending algorithm will place on the trending list an average of 90 minutes before they get there, and up to four or five hours in advance in some cases.
How It Works
Twitter’s own trending algorithm selects topics automatically based on overall number of tweets and recent changes in tweet volume. Shah and Nikolov’s algorithm works based on the “machine learning” principle of taking a sample “training set” of Twitter data that both trended and did not trend and searching for patterns. The algorithm tracks changes in the number of tweets about new topics to changes in the number of tweets about every topic in the sample set, with sample topics that are similar to the new topic weighted more heavily. Based on how sample data performs in comparison to new data, the algorithm creates an estimate of whether new data will trend.
Shah and Nikolov say the algorithm can be scaled according to computational resources (meaning it can be divided among multiple servers for greater processing power) and also becomes more accurate as the size of the training set of data increases. In experiments with a training set of Twitter topics that did trend and 200 that didn't, the algorithm predicted trending topics among live tweets with 95% accuracy and a 4% false positive rate.
Taking Advantage of Trending Topics
For Twitter, the new algorithm developed at MIT could potentially allow the social media provider to charge a premium for ads associated with developing trending topics. For digital marketers, advance knowledge of trending topics could greatly assist efforts to perform real-time social marketing and ensure that Twitter promotional messages are timed ahead of trending topics, rather than developed in response to them.
In addition, Shah and Nikolov say the algorithm can be applied to any set of data that follows trends and varies over time, such as stock prices. It could also be applied to detect content trends on other social networks such as Facebook and Google+.
Shah and Nikolov are not the only ones attempting to forecast what’s hot in the world of Twitter. A recently launched service called TWeather is designed to give reports on what is emerging on Twitter -- the “clouds and patterns” of tweets on different topics, providing 10-minute “weather” updates on popular trending topics in Twitter. From the home page, users can select any one of the top 10 currently trending topics or a link within featured topics such as politics and music. From there, the user is brought to a “TWeather report” that displays a cloud of constantly shifting keywords relating to the topic. The larger and closer to the right of the screen a keyword is located, the more “hot” it currently is.
By selecting a keyword, users can see related tweets. Reports are updated every 10 minutes and users can scroll back to see previous reports. Thus TWeather presents tweet trending data in a way that makes it easier for users to detect and track patterns than they can by scrolling through topical Twitter timelines. Users can subscribe to TWeather reports from their Twitter accounts, and once a user subscribes to five different reports, they can create their own TWeather reports.
BostInno interestingly points out that a marketer using Shah and Nikolov’s algorithm might have particular success in applying it to targeting “hipsters,” the young urban demographic that specializes in irony and keeping up with the very latest trends. BostInno points out that the algorithm successfully predicted the popularity of #NoShaveNovember, a hipster-friendly topic about a month long facial hair contest that is currently trending on Twitter in Boston, hours before it popped up.
“This sounds like an ideal algorithm for the hipsters out there,” states BostonInno. “They can discover what’s cool before it is cool. Just know, Twitterverse, once the trend surfaces, the hipsters will be on to the next underground fad.”