Big data, data analytics and the rise of the machines!

The rise of the machines immediately paints an apocalyptic image in the mind – mainly thanks to the Terminator franchise – but today our world is dominated by machines. Our PC’s, laptops, and mobiles devices. Our TV’s, cars and the Internet of Things.

But machines are even more prevalent when it comes to data, especially big data. We – and of course the machines – are generating so much data that powerful computing machines are required to process and store it.

What is quite scary – depending on your point of view – is that machines are learning. Through computer algorithms they automatically upgrade themselves by discovering patterns in existing data without being explicitly programmed.

This whole processing of machine learning, rather than being sinister future vision, is now a reality and a requirement in dealing with the mountains of data we are producing. Essentially, many of our systems in business and the wider society wouldn’t work without data, machines to process the data and of course, humans (data analysts) to make sense of it all.

5 crazy facts about big data

To give a sense of how mind-boggling the numbers are when we talk about data and the growth of the big data phenomenon, here are five facts that may make your brain hurt! 

  1. What is 2,500,000,000,000,000,000? Well, it is two and a half Quintillion. Never heard of it?  Most haven’t, but this is what users of the internet generate in bytes of data each day, on average.
  2. What platform allows the sending of 65 billion messages per day? It’s WhatsApp. They have 2 billion users and over 5 million companies using the platform.
  3. The global finance sector has increased it’s spend on big data infrastructure by $18 billion in five years. What is it worth now? Over $30 billion!
  4. How long would it take a person to download all the data from the internet? 181 MILLION years. Now if you turned this into miles it would take you 15 minutes to cover that distance if you travelled at the speed of light.
  5. What percentage of films are watched by users based on Netflix’s machine learning algorithms? Approximately 75% of Netflix users select films recommended to them by the company’s machines.   

Data is sexy

In 2012 the Harvard Business Review hailed data science as ‘the sexiest job of the 21st century’ and this seemed like an exaggerated view of those working in the world of data. Fast forward to today and every business wants to employ data scientists. The value of data, or more accurately, the value of the insights it offers have been realised and companies want data scientists. The field of data science covers many disciplines, data analysis, informatics, AI, Machine learning, numerical analysis, words analysis, business analytics, and plotting.

More than anything, what data scientists do is make discoveries from data. They are comfortable in the digital economy and able to bring structure to large quantities of formless data.  In a competitive landscape where data never stops flowing, data professionals help decision makers shift their focus to the ongoing customer signals that come from data.

The data analyst is on the rise as businesses appreciate the challenges they face to catch and keep up with volumes of data. Forbes reported that 95% of businesses cite the need to manage unstructured data as a problem for their business. And, the CIO highlighted that  80-90% of the data we generate today is unstructured.

The challenge has been accelerated with faster mobile networks leading to increased usage, IBM reports that 90% of all data has been created in the last two years.

This all supports the argument that skilled data professionals and more computing machine power will be required as we move into the 2020s. It is predicted that 97.2% of organizations are investing in big data, and AI and job listings for data science and analytics will reach around 2.7 million by 2020, according to recent research by Forbes. 

Gaining insights from data

Data science extracts knowledge and insights from data, often information that would not be available from just one set of data. The term “big data” refers to data that is so large, fast or complex that it’s difficult or impossible to process using traditional methods.

The roots of data science are in scientific methods and algorithms and these are rigorously used. The output has to be verified as plausible. Key aspects are based on the ability to evaluate extremely large data sets that may be analysed computationally to reveal patterns, trends, and associations, especially relating to human behaviour and interactions.

Companies are competing in the data capture race. According to Globe Newswire, at the end of 2019, worldwide spending on Big Data was already worth $180 billion, and it is projected to grow at a CAGR of 13.2% between 2020 and 2022. Reports have it that IT purchases, hardware purchases, and business services could receive the highest spending on big data analytics.

The aim is to better understand customers. Data gives direction on what customers need and want.

Netflix, for example, is using data to nurture customer loyalty. The company saves £$1 billion per year in customer retention and over 100 million subscribers. Collecting huge amounts of data from users is key to their retention strategy. If you are a subscriber, you are familiar with how they send you suggestions of the next movie you should watch. This is done using past search and watch data – key insights on what interests the subscriber most.

For many companies, consumer data offers a way to better understand customer behaviour and to improve the customer experience. Data such as reviews and feedback, are used to nimbly modify their digital presence, goods, or services to better suit the current marketplace. Just think Amazon – retail, cloud computing, advertising and a continuous development of products to capture different markets.

Contextualized data can help companies understand how consumers are engaging with and responding to their marketing campaigns, and adjust accordingly. This highly predictive use gives businesses an idea of what consumers want based on what they have already done. Mapping users’ journeys and personalizing their journey, segmenting data effectively means companies market to the people more likely to engage. This helps to transform the data into cash flow.

For businesses that capture large amounts of data, collecting information and then selling it, represents opportunities for new revenue streams. Data brokers, or data service providers that buy and sell information on customers. This subset of the data boom has risen alongside the big data phenomenon. Some businesses even use consumer data as a means of securing more sensitive information. For example, banking institutions sometimes use voice recognition data to authorize a user to access their financial information or protect them for fraudulent attempts to steal their information.

So, what does the future hold – do the machines keep rising?

In short, yes. We love generating data in the value exchange with corporations to use their services, and corporations love using data to help them make decisions. In their 2020 blog “The future of big data 5 predictions from the experts’ 2020-2025”, Itransition has some interesting predictions:

  • Machine learning will continue to change the landscape – Experts believe that computers’ ability to learn from data will improve considerably due to more advanced unsupervised algorithms, deeper personalization, and cognitive services. As a result, there will be machines that are more intelligent and capable of reading emotions, drive cars, explore the space, and treat patients. Investment is huge.
  • Data scientists and CDOs will be in high demand – The positions of Data Scientists and Chief Data Officers (CDOs) are relatively new, but the need for these specialists on the labour market is already high. In 2019, KPMG surveyed 3,600 CIOs and technology executives from 108 countries and found out that 67% of them struggled with skill shortages (which were all-time high since 2008), with the top three scarcest skills being big data/analytics, security, and AI.

Being frightening and fascinating at the same time, the future of big data analytics promises to change the way businesses operate in finance, healthcare, manufacturing, and other industries.

The overwhelming size of big data may create additional challenges in the future, including data privacy and security risks, shortage of data professionals, and difficulties in data storage and processing.

However, most agree that big data will mean big value. It will give rise to new job categories and even entire departments responsible for data management in large organisations. This means that if an organisation is to benefit from data they need to find the right skilled data specialists, and quickly!

So there seems no end to the rise of data and machines in every facet of our lives where third parties control our privacy, choices, protection of our information, news, and social influences. Is this a welcome development or will data scientists and their machines become the controllers?


We will look at the rise of data and how employers can develop data analytics skills in our live webinar: Data Analytics – Rise of the Data Analyst