As artificial intelligence and digital transformation continue to dominate discussions this year, data management firms watch on with the knowledge that the path from data collection to meaningful use remains far from straightforward.
According to the UK Business Data Survey, while nearly all the companies surveyed report handling digitised data, only just over 20% actively analyse it for new insights or knowledge. Of that group, only a third say this leads to innovation or the creation of new business functions.
Behind the scenes, experts highlight that most companies still face significant challenges in turning raw data into valuable insights.
While data is abundant, it is often fragmented, inconsistently structured, and of variable quality. Many companies have yet to define clear strategies for its use.
The promise of data extends beyond merely informing AI capabilities; it drives innovations such as digital twins, which improve operational efficiency, reduce carbon emissions, and enhance preparedness for cyberattacks. Yet, many businesses still struggle with harnessing the potential of this resource.
“One of the biggest challenges right now is that companies have so much data—structured, semi-structured, and unstructured—that it can be overwhelming,” says Vishal Marria, CEO of UK-based data management firm Quantexa.
Marcin Figurski, technical director at Google Cloud consultancy Qodea, echoes this sentiment, noting that fragmentation and data quality are ongoing issues.
“Another challenge is getting the right people to handle the data properly. It’s easy to gather data, but not easy to organise and use it effectively,” he explains.
He adds that many companies simply store data without categorising or structuring it, often with the vague hope that it might be useful later.
“But this approach doesn’t lead to actionable insights,” Figurski says.
“The problem is that they didn’t even know what data they had,” Figurski adds, highlighting the complexity created by a “shadow IT” environment in which different departments hoard and handle data independently, often without a centralised governance framework.
“There’s a lot of hype around ‘every company should be a data-driven company.’ But people assume that simply collecting data will lead to valuable insights. That’s not the case. The challenge is understanding the purpose of data, structuring it properly, and knowing how to use it effectively,” says Figurski.
Chris Harris, VP of field engineering at database platform vendor Couchbase concurs, pointing out that most businesses are sitting on vast, untapped data resources without the means to extract their full value.
“It’s like sitting on a gold mine without any shovels,” he says.
Without addressing these issues, Harris warns, companies risk falling behind in an AI-driven landscape. He adds that most organisations currently lack a suitable data strategy for generative AI, and only a fraction have implemented the necessary tools for advanced AI applications.
“The path forward requires strategic investment in robust data strategies, including infrastructure, skills, and culture. This isn’t about merely keeping pace—it’s about setting the stage for innovation in an increasingly data-centric business world,” he argues.
Reflecting on the past year, Marria notes that many companies started running pilots and proofs-of-concept when generative AI became a hot topic, but the fundamental challenge always came back to data: “How to make it accessible, contextualised, and trustworthy—this is where many pilots faltered,” he says.
“The key to success is curating data properly so that it can be used effectively with AI.”
Harris adds that business data must be well-organised, seamlessly integrated, and readily accessible. “It needs to be accurate, consistent, and compliant with regulatory requirements. Achieving this level of data management requires organisations to carefully balance data acquisition, storage, and processing methods.”
“Establishing this solid data foundation is essential not only for generating meaningful insights but also for enabling modern applications that rely on data,” he concludes.
For Quantexa, helping organisations manage their data has led to significant outcomes, such as improvements in fraud detection and customer experience.
One notable case was during the UK’s Covid-19 lockdowns, when the firm partnered with the UK Cabinet Office to support the ‘Bounce Back Loan Scheme’, designed to support small and midsize businesses by loaning between £2,000 and £50,000 at a low interest rate.
However, the government estimated that around 7.5% of its loans were fraudulent and needed a quick solution that could ingest more than 100 million data items and analyse it for dishonest behaviour.
“The key challenge was handling large datasets from multiple sources—external and internal data, and application data from various organisations—all without a common structure,” Marria explains.
The goal was to unify and contextualise the data, then apply predictive AI to detect anomalies, understand patterns, and drive action.
Similarly, Quantexa worked with telco giant Vodafone to optimise customer data management and improve customer service. “With the right data interpretation and AI, it becomes easier to identify customer issues, manage complaints, and better serve the customer,” says Marria.
The company also partnered with the bank, HSBC, where Marria notes, “it’s done a great job in getting their data in order. Now, we’re helping them take it to the next level by integrating LLMs with trusted, contextualised data and using knowledge graphs to drive deeper insights.”
Meanwhile, Qodea’s Figurski argues that many companies are missing out on the potential of digital twins. “First, we’d need to understand the use case. What do they want to achieve with the digital twin? Is it forecasting, modelling, optimisation, or something else?” he explains.
Once the desired outcome is clear, the next step is identifying the data needed and how to access it. “After centralising the data into a system, we can begin to build a digital twin model based on the specific needs,” says Figurski.
From there, the process moves through proof of concept, testing, and eventual scaling into production.
Digital twins can be applied in a variety of ways. For instance, in cybersecurity, they allow companies to simulate attack paths and test their defence systems. In supply chain management, digital twins are used for predictive pricing—modelling the impact of geopolitical instability or climate change on supply chain disruptions.
“For example, businesses might use a digital twin to model and predict the impact of supply chain disruptions on prices,” Figurski explains.
In the realm of fleet optimisation, digital twins are helping companies model vehicle operations to reduce costs by improving driving behaviours, such as reducing harsh braking, optimising fuel efficiency, and refining delivery routes.
On the environmental front, digital twins are also being used to track and reduce carbon footprints. By simulating supply chain processes, businesses can measure the environmental impact of sourcing materials and assess their overall carbon emissions.
According to Marria, for organising the data AI is proving helpful.
“AI is a game-changer for intelligent data management. It allows organisations to make sense of their data in real-time, with full context—particularly valuable in regulated industries,” he says.
His firm also debuted its own GenAI tool, Q Assist, this summer that helps its customer’s teams get helpful insights into their data, backed up with Quantexa’s information base.
“The idea is that the LLM takes feedback, processes it, and integrates it with reliable data sources to create a meaningful interaction,” says Marria.
He adds that HSBC is using it to streamline analysis, reduce reliance on data science teams for ad-hoc data requests, and also support customer-facing teams with data insights.
Marria believes that companies using AI incrementally for efficiency alone risk missing the bigger picture.
“AI is about rethinking operations, not just speeding up old processes. It’s like transitioning from horse-drawn carriages to cars—you can’t just make the carriage faster; you need a completely new way of thinking.”
However, Couchbase’s Harris emphasises that AI may require a redesign of data architecture.
“Future-proofed organisations will need to consolidate platforms, eliminate silos, and create data systems that capture the ‘intelligence history’ of AI,” he says, highlighting the need for transparency and governance.
While AI offers immense promise, experts agree that its true potential can only be realised once organisations overcome the foundational data challenges.
As Marria concludes, “The next 10 years will be a transformative period, with many more use cases emerging and more benefits to be realised. But it’s going to require a lot of technical innovation to keep pace.”
Read more here: Shaping AI’s future: can the world agree on regulation?
Subscribe to our Editor's weekly newsletter
Share
