Retailers now have unprecedented access to consumer data, but face challenges putting this big data into action. By identifying and understanding some of the challenges in this area, retailers can better decide where to focus their efforts. 

Below are some common use cases for big data in retail and the associated challenges. We've skipped the issues caused by the very nature of big data, such as fighting breakneck data growth or processing unstructured data. We also skipped challenges that can be easily solved by choosing the right technology, such as challenges related to generating insights in a timely manner.

The following three use cases are by no means exhaustive, but do represent where many retailers are currently channeling their energy.

Creating a 360-Degree Customer View

Challenge: Recognizing a customer in different channels

Creating a 360-degree customer view is a challenge in our current omnichannel world. Retailers should know that the visitor who was surfing the online catalog yesterday, the customer who made a purchase at their brick-and-mortar store a week ago and the person who is talking to their call center right now is, in fact, the same person.

Let's compare a customer’s journey with a puzzle, where each element is that customer’s particular touch point with the retailer. The more elements the retailer has, the easier it is to guess what the picture is. But the retailer has to deal with millions of customers, and correspondingly, millions of puzzles. Naturally, the retailer is unlikely to put each puzzle together if they even fail to identify which element belongs where. 

The solution to this big data challenge is to first recognize a customer at every touch point and then to collect data about the customer’s journey and link it to the customer’s ID.

How you identify a customer varies from one channel to another. For digital ones, such as online stores or mobile apps, visitors share some personal information (at least, a name and an email) to register. Unfortunately visitors sometimes game the system by using an imaginary name or indicating an email they rarely check. Providing a social login option often serves to encourage visitors to share real data. Visitors appreciate the convenience and the retailer will increase the chances of learning who the visitor is.

In physical stores, a customer can just walk in and walk out, and a retailer will never know who the customer was. However, brick-and-mortar retailers are also striving to find solutions for this and new (and old) tools are helping, including loyalty cards, beacons, mobile apps.

Dynamic Pricing

Challenge: Optimizing prices with no harm to KPIs

Big data consultants often describe dynamic pricing as if it can work miracles. Indeed, the concept is fascinating: an analytical system automatically monitors the competitors’ prices and updates its prices based on the defined rules. Do you want to be always 5 percent cheaper than your rivals? Your wish may come true!

Learning Opportunities

However, dynamic pricing only goes so far. After all the changes, the prices should still align with the retailer’s pricing strategy and shouldn't ruin sales, margin and profit KPIs in the process. No retailer wants to end up with negative margin due to price wars. Besides, if one of the competitors has a particular SKU on promotion, trying to beat their promo price with a standard one will be unreasonable.

To overcome this challenge, a retailer should take on dynamic pricing only when they have developed a clear pricing strategy and a well thought out hierarchy of KPIs. To avoid negative margins, retailers should set certain thresholds: for example, the margin should never drop below 5 percent. Additionally, the analytical system should be tuned to check how a price change influences revenue, margin and profit. And definitely train the system to identify outliers such as promo items and exclude them from the analysis.

Making Personalized Recommendations

Challenge: Retraining the recommendation engine

Usually, retailers choose a mix of content-based and collaborative filtering techniques for their recommendation engines. The first approach ensures the engine knows what products each user liked or disliked in the past, while the other approach complements it with insights into all users’ behavior and preferences. The task of the recommendation engine is then to predict what products a user might like.

As the engine has to deal with big data, there is no other option than generating recommendations automatically, which becomes possible thanks to machine learning. However, even after the ‘training’ on historical data is completed, retailers should not expect the engine to show seamless performance and 100 percent accurate recommendations. New customers are especially at risk of being irritated rather than pleased with irrelevant product offers, as the engine does not yet know their histories.

To improve accuracy, the engine should have a control loop that checks the relevance of generated recommendations based on customers’ response rate. For instance, the website visitor viewed four out of five recommended products and ignored the fifth one. If other customers with a similar behavior pattern have skipped the same recommendation, the engine should make an adjustment.

Preparation Is Half the Battle

While retailers face some real challenges with big data, this isn't to say they shouldn't pursue these projects. Understanding the real picture before embarking on a big data project helps retailers understand the effort and time it takes to successfully see these projects through. Retailers still need to adress challenges even before the project starts, such as setting up a KPI hierarchy and defining price strategy prior to implementing dynamic pricing. 

Forewarned is forearmed. Knowing the challenges in advance, you’ll be better prepared to solve them and make your project a success.