Request a demo

Fill in the form below and we'll get in touch within 2 working days.

[contact-form-7 id="5268" title="Request a Demo"]

Dark data – friend or foe?

Estimated reading time: 3 minutes, 54 seconds

Dark data has for some time been the buzz word on the data block, dropped into conversations and delivered with a tone that puts it on a par with being related to the likes of Darth Vader or Bane. Yet many businesses are unsure of what dark data means and whether or not it’s cause for concern.

So what exactly is dark data? Put simply, it constitutes all of the unknown data within an organisation. It is a subset of big data that makes up the largest volume of information collected by organisations each year. It’s untapped, unstructured and is normally found within data repositories having not been analysed, processed or cleaned.

According to a recent study carried out by Splunk, 55% of all data globally is unquantified, rising to 75% when it comes to businesses. Many either struggle to find the data in the first place, do not have the inclination to try to make sense of it or simply lack the resource and commitment to locate and quantify it.

A new Veritas study only furthers this issue. It found that half of company data is unclassified and businesses have limited or no visibility over vast volumes of potentially business-critical information, pointing to an increased risk of security breaches. Businesses need to safeguard against potential security breaches, so using and making sense of as much dark data as possible is the best course of action to achieve this.

However, there are several obstacles when it comes to overcoming dark data within businesses that are, fundamentally, team issues. The first and possibly the most problematic is volume of data. Although businesses have some understanding of the value that data can have, they are overwhelmed with the volume of it and do not have the resource to tackle it. As much as a simple solution would be to train more employees in how to analyse data, many businesses don’t have the budget or don’t want to expend their time on data analytics training.

With such a fast evolving, technologically-driven business world, this is overwhelming. It demonstrates how data strategy – and making sense of data – has to be regarded as an essential part of a business for it to have an effect at all. Businesses need to be educated on the value of all their data and the hidden insights that it can provide because, after all, the benefit of having the right data at the right time can be extremely advantageous.

In today’s data-rich landscape, any business that can extract the true value from their internal and external data to make informed decisions is at a huge advantage. For businesses that have an absence of employees trained in data – and to make the solution of tackling cyber security issues simpler – they shouldn’t be afraid to use machine learning and AI.

Data intelligence platforms can be used to help quickly clean large volumes of data from different sources and make sense of it, which is necessary to avoid security breaches. However, by employing these advanced techniques on dark data, businesses will also uncover hidden patterns and insights.

In fact, CBR suggests that between 61% and 67% of organisations see value in AI technologies which would enable them to look more closely at insights in operational efficiency, innovation, strategic decision-making, recruitment, and customer experience.

In the sectors we currently work with, including oil and gas, mining and utilities, our own data intelligence platform, Ossian, is used to simplify and overcome a wide variety of issues, from improving supply chain management to identifying gas flaring and locating off-grid mines. The wide variety of data that AI can make sense of only goes to show how beneficial data intelligence platforms can be for businesses when it comes to creating competitive advantage for themselves.

As the volume of dark data within organisations increases, the potential to miss opportunities through untapped insights is growing. It is therefore imperative for organisations to be assured that, although dark data can often be associated with missed information and cyber-security threats, through overcoming obstacles and training employees to handle the data, it can produce valuable information which can be translated into viable business opportunities.

Furthermore, as businesses adopt a more open approach to disruptive data analytics and machine learning, they can open up the possibility of harnessing competitive advantage through information they didn’t even know they had, helping them to make faster, more efficient business decisions before the competition has even got out of bed. It’s time to turn the light on dark data so we can start seeing it as a friend rather than a foe.

First published 29th July on DataIQ.

Posted on July 31st 2019 in Press & PR

  • Brainnwave has been instrumental in helping UNICEF understand how people move so we can coordinate relief assistance in a much better, more targeted way and can assist the population where the need is greatest.

    Natalia Adler – Data, Research and Policy Planning Specialist – UNICEF

    Read case study

  • Brainnwave are a great team to work with. They put a focus on working collaboratively with us and they make sure that they’ve always understood what we need.

    Their solution has already saved a lot of time through the automation of organising, updating and sharing some of our day to day data (such as fuel prices) across the organisation.

    Noreen Howat – Senior Strategy Manager – Aggreko

    Read case study

  • Brainnwave achieved more in 2 months with 4 analysts than a tier 1 consultancy achieved in 6 months with 6 people.

    Shawn Pace – Global Sourcing Director – Goodyear

  • Brainnwave are helping change the way we do business.

    The platform Brainnwave are developing is using data analytics to provide us with the insights that we cannot access today due to the complexity of the questions we need to answer. This will be a game changer for our global business.

    Stéphane le Corre – Strategy & Commercial Development Director – Aggreko

    Read case study

  • I have always been a visual thinker and Brainnwave’s Ossian platform enables me to see things that were previously buried deep within reports; this helps me to react quickly to the business environment and make decisions with confidence.

    Brian Chalmers – Strategy and Development Director – Wood plc

  • I have been championing, for some time now, the need to bring digital innovation from other industries into the housing sector.

    We are excited to be working with Brainnwave to apply their data intelligence technology to the housing sector. Their innovative approach to data and analytics, developing deep insights about our customers, has never been done before and will help us provide the best possible customer experience.

    Doug Silverstone – Head of Data & Analytics – MTVH

  • We are delighted to be working with our Partner, Brainnwave, on their new product, Geo Spatial Web Services. Brainnwave’s new product provides efficient access to our geospatial data providing users with the ability to analyse the data and generate strategic insights to aid their business.

    Stefan Wells – Head of Partners – Ordnance Survey

  • Brainnwave are a great example of how companies can deliver complex data science projects by leveraging their network, partnering with academia and by building the ecosystem of data experts.

    The Data Lab has been a partner with Brainnwave for several years and they have contributed greatly to the data driven innovation agenda for Scotland and have leveraged opportunities to build and grow their data talent and expand their capabilities significantly in both data science and artificial intelligence.

    Gillian Docherty – CEO – The Data Lab