Big data held the title of "The Next Big Thing" for quite a while. Tech publications and mainstream media alike showered attention on it and we were all promised that if only we could treat the data we gathered, we would find solutions to practically every problem imaginable.
The problem with big data was that a lot of Big Promises were made, with a Big Lack of real world applications. Was it all hype, or would it actually turn into actionable insights that make the world a better place?
A good place to start answering this is Gartner’s annual hype cycle. For those who don't know the hype cycle, Gartner's yearly reports show where new technologies fall on a scale from Technology Trigger to the Peak of Inflated Expectations to the (brilliantly named) Trough of Disillusionment. If the technology actually has legs, we can expect it to mount through the (equally well-named) Slope of enlightenment before reaching tech Nirvana at the Plateau of Productivity.
In 2014, big data was en-route to the Trough of disillusionment (uh-oh!), yet in 2015 it is nowhere to be found on the hype cycle. What could this mean? Presuming Gartner didn’t forget about its existence, this might just mean that big data is now beyond the hype and is being used in the real world.
However, if you’re still confused about how companies are actually turning big data into insights and action, you’re not alone. Hopefully the following case studies will give you a better idea:
1. Like Tinder, for Cows
Cattle farmer's priority is producing beef and milk. And most don't want to leave calf production to chance or true love, so they turn to artificial insemination. To increase chances of inseminating cattle, they need to ‘catch’ cows when they’re in heat (roughly one 14-hour period every 21 days and usually late at night). If the timing is right, the chances of successfully inseminating the cow are around 70 percent, with a 40 percent chance of this leading to a pregnancy.
In Japan, cattle farmers were pretty good at detecting when their cows were in heat — getting it right about 55 percent of the time. They noticed the cows stamped around more during the time they were in heat. To increase their 55 percent rate, the farmers contacted Fujitsu to see if a data-driven model could improve their chances.
Fujitsu attached movement detection monitors to the cattle and hooked these up to a cloud system based on Azure. They were then able to notice the three-weekly peaks in each cow’s movement and predict close to 100 percent of the time when cows were in heat. Farmers received a notification on their mobile and never missed an opportunity to inseminate their stock. This caused a leap in detection rates — from 55 percent to 95 percent and meant successful pregnancy rates increased from 40 percent to 67 percent.
2. Helping Doctors Tackle Ebola in West Africa
There’s been a lot of excitement around big data in the healthcare industry. From predicting flu outbreaks, to giving patients more personalized care, big data promises to change the face of the industry. A specific case where it’s already been used is to help contain the spread of Ebola, the deadly disease which has killed close to 5000 people in West Africa since late 2013 (and which is thankfully being beaten).
Researchers at the Mailman School of Public Health used real time data from the World Health Organization to publish forecasts with the cumulative infection rate and mortality rate up to six weeks into the future. The team were able to build statistical algorithms to create a simulation based on Ebola’s incubation period. This model was updated daily online and helped doctors prepare for the spread of the disease, ensure trained staff were in place and make sure there were enough beds available for victims.
3. Harder, Faster, Better, Stronger
Statisticians and sportspeople might not seem the most obvious bed fellows, yet big data is being used extensively in the world of professional athletics to improve sports performance. While a home advantage (and we wouldn’t want to admit it, but also a dependence on obscure sports) helped, big data helped the United Kingdom into third place on the medals table at the 2012 Olympics — a big improvement on the average for previous years.
For example, the British Cycling Team used performance analysis tools to better understand almost every part of their approach to racing. The data highlighted weak points in their methods and radically improved their medal gains at competition.
4. Clean, Green and Big
A major obstacle preventing wind and solar energy success is that providers struggle to assure a stream of always-on electricity. If the wind drops or you get a particularly cloudy day, production drops to a standstill. This means that energy providers always need to be able to provide a backup of fossil fuel generators, in case green production stops. However, you can’t just turn a coal power plant on — it needs to heat up first.
India is making big progress in renewable energy provision, yet has been struggling with this exact issue. However, big data is helping energy providers predict and prepare for variations in renewable energy production. By analyzing real time weather patterns, historical data and information about technical features of wind and solar plants, energy companies are getting much better at predicting the amount of "back-up" energy they’ll need that day from fossil fuel based generators.
5. Card Trick
It’s no secret that major TV and movie streaming providers like Netflix and Amazon use big data to analyze and predict what original content their viewers will find popular. Reportedly, the hit Netflix show House of Cards was — in part — commissioned off the back of big data analytics.
Pivotal.io, a Big Data analytics provider, released a case study earlier this year about a commissioned project to help create the ultimate hit TV show using big data. The team analyzed ratings figures, manually collected metadata and show transcripts. After cleaning up this (rather messy) information, the team were able to provide insights into which features would lead to the highest ratings, including variables such as:
- Speaker characteristics
- Number of people shown on screen at a time
- Broadcast topics
Big data has moved beyond the hype, and is providing real insights to professionals in a range of industries. However, while big data has a lot to offer, it’s not a be-all and end-all. As a recent interview with a data analyst pointed out: “You can measure absolutely everything, but you need to make sure there’s rationale behind what you’re collecting.” Of the tens of thousands of lines of data they collect [in sports], it is then condensed into only two to three digestible and pertinent points that are fed to the athlete during competition time. Anything more could create information overload, “this is the true art of the analyst to make this information edible for the coach and analyst.”