In today’s digital world, we generate an enormous amount of data every day through online activities, such as social media posts, emails, transactions, search queries, and more. This massive volume of data is called big data. The term ‘big data’ refers to data sets that are so large and complex that traditional data processing tools and techniques are unable to handle them effectively. Big data has revolutionised many different industries.

Big data is often defined by the 3 V’s: volume, velocity and variety. ‘Volume’ is characterised by the sheer amount of data that is generated every second from various sources such as social media, sensors and devices. ‘Velocity’ refers to the speed at which data is generated and the need to process it in real-time; like in stock market data, weather updates and keyword bidding. ‘Variety’ implies the various forms of data: structured data (databases), semi-structured data (excel sheets) and unstructured data (social media posts). 

The evolution of big data has been rapid and ongoing since the term was first coined in the early 2000s. Initially, the focus was on volume; how to collect and store vast amounts of data. This began as a distributed computing framework (Hadoop) and emerged into cloud computing to provide a flexible and scalable way to store and process big data. This then solved ‘variety’ as it allowed the rise of the ‘Internet of Things’ – a network of connected devices that allows for technologies like smart homes. 

As the complexity of data grew, machine learning and AI became more important in making sense of it all. Ultimately, all this is leading towards real-time analytics, which enable businesses to analyse data as it is generated; solving ‘velocity’. Naturally, all this application of data has led to concerns about data privacy and security. Regulations like the EU’s General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) were introduced to provide transparency regarding an organisation’s data collection and usage purposes – limiting the amount of data organisations can collect and use.

How does the evolution of big data impact marketing? Here are a few thoughts below.

Be Pragmatic About Hyper-Personalisation

Personalisation has become a buzzword in marketing, and for a good reason. Customer experience has become a key battleground for organisations and the vision was to win by building unique interactions across touchpoints. Big data was key to making that happen as more and more data began to be collected, marketers envisioned accurate and detailed customer profiles which would enable them to deliver tailored content. However, the reality is that the ‘segment of one’ is likely never happening. In 2021, Gartner found that 63% of marketers struggle with personalisation. Even if you figure out seamless implementation issues, users are no longer allowing access to data, especially to be shared with third-party platforms. This limits the effectiveness of personalisation as it becomes a challenge to link behaviours and attitudes back to one person. As a result, ‘hyper’ personalisation is more realistically applied on clusters of customers, the aim being to micro-segment your customer until you reach the elusive ‘segment of one’. So marketers, for now, envision tailored offers and recommendations for each customer cluster’s specific needs and preferences.

KPI, Metrics and Analysis Will Come and Go

Les Binet’s recent thoughts on how attribution will be replaced by econometrics reminded me of my favourite axiom – Goodhart’s Law, which says “When a measure becomes a target, it ceases to be a good measure.” There have been many examples in recent history where founders of metrics have cautioned against their usage. The economist who is attributed with inventing the modern version of GDP, Simon Kuznets, famously cautioned that “the welfare of a nation can scarcely be inferred from a measure of national income” as GDP evolved from a metric meant to measure the capacity of US industry to manufacture weapons for WWII to one that denotes economic prosperity. Measuring is messy and convoluted, especially for any qualitative or descriptive variable (like human development or customer satisfaction). Marketers tend to love silver bullet metrics that can simplify things. However, it is critical to understand the limitations of any metric or analysis in order to make meaningful interpretations from it: attribution and GDP are great guiding KPIs if used sensibly. Behaviours and trends can only be determined by combining multiple and seemingly disparate data points – the big picture is always a result of ‘multiple sources of truth.’

Ask the Right Questions

The advent of AI chatbots, like ChatGPT, has left many wondering about the future of many job functions, even within analytics. Personally, I am not sure about how it impacts roles but what I can sense is that marketers need to be able to ask the right questions. The volume and variety of data sources now available have allowed marketers to have a peek into many more touchpoints across their customer journey. However, condensing this data into insights remains a challenge – we can often tell you ‘what’ but struggle with the ‘why’. The best way is to inculcate a hypothesis-testing approach. There is no ‘right’ or ‘wrong’ way to go about it, you can start top-down or bottom-up, you can start with first-party metrics or even anecdotal evidence you heard or observed. The critical bit is the ability to pivot your thoughts and keep the feedback loop going in order to refine your hypothesis. Tools like ChatGPT make it even easier to access information, but the more specific the questions you ask it, the more specific answer you will get from it. At the end of the day, a human has to infer the information that AI analyses.

Incentivise Information

Consumers are acutely aware of becoming ‘data products’ thanks to new-found concerns over data privacy. According to Gartner, the use of ethical AI is a top concern for 70% of CMOs – in terms of bringing accountability in the use of AI-based marketing. But customers also want to leverage their information and Gartner found that 25% of consumers would allow tracking if they are familiar with the brand or publisher requesting the tracking, especially as part of an explicit value exchange, such as cash rewards, coupons, discounts or loyalty points. 

The domain of analytics is moving in a similar direction. As tracking becomes limited for entities in their first-party data, they will move on to third-party aggregators. I personally feel that survey-based feedback is poised to make a big comeback, albeit not necessarily via traditional market research methods. Similarly, the rise of ‘clean rooms’, data platforms that host aggregated and anonymised user information to protect user privacy, will be used to provide advertisers with non-identifiable information to target a specific demographic and for audience measurement.

Ans Khurram is an analytics and insights professional. He was included in the Insights250 List 2022 and the 30 Under 30: Rising Stars of Business Analytics List 2021.
anskhurram@gmail.com