What Are Data Platforms, and How Do You Build a Modern One?

What Are the Characteristics of a Modern Data Platform?

Modern data platforms can gather information from a variety of sources into a centralized medium, such as a data lake. The data is then interpreted for business intelligence applications.

According to Raihan, a modern data platform is composed of three key components:

• An analytics layer, which Raihan says “enables easy querying of data so practitioners who are subject-matter experts can ask relevant questions in a business context”
• A search component, which “enables federation of data across multiple data sources, allowing for a deeper level of analytics in the layer above” 
• A storage layer, which allows data to be organized by age and business value

All of this data is ingested from a variety of tools, such as Internet of Things devices. This data may have existed previously in separate departments, but such silos are now less desirable, according to ESG’s Catanzano.

“People are realizing that data needs to be real-time in a lot of cases to get to decision-making,” he says. “So, streaming is really picking up just to get to quality data as quickly as possible. And there are some great advantages once you get there.”

DIG DEEPER: Learn how growing businesses are improving their data performance.

Data Platform Building Blocks: Agile, DevOps and DataOps

Modern and enterprise data platforms are built on two key methodologies of modern software development: DevOps and agile development. Though similar in nature, the process of agile development focuses on quick iteration while DevOps brings development processes to operations.

At the center of this discussion is the concept of DataOps, an operations-driven approach to data. Modern data platforms fit neatly into the DataOps concept because, as Gartner explains, it allows for collaborative data management and automated flows between data manager and data consumers.

Click the banner below to learn how a modern data platform supports smart decision making.

Data Platform Building Blocks: Data Ingestion and Data Pipelines

DataOps can consume a wide range of data from any kind of source, which makes data pipelines a core component of modern data platforms. These are the delivery mechanisms that make this technology possible.

According to IBM, data pipelines can take different forms. They can be delivered in batches or in real time, and as structured or unstructured data. The difference? Batch data may be delivered once a day, while real-time data is delivered as it happens.

The process of centralizing that data in real time, called data ingestion, is also important to modern data platforms. As TechRepublic notes, data ingestion relies on automation to properly function, and may combine real-time and batch processes.

EXPLORE: How businesses can save millions with more efficient data centers.

But while data ingestion is key to many enterprise and modern data platforms, the way that data is processed isn’t always the same. Splunk’s Raihan observes that data ingestion is occurring more within edge computing contexts, which makes it easier to perform data processing onsite.

“The other trend that’s relevant here is the expansion of edge deployments where data is ingested as well as processed,” he says. “A greater amount of analytics is being performed at the edge to enable real-time action even before it reaches an on-prem data center or cloud.”

Data Platform Building Blocks: Data Lakes vs. Data Mesh

Data platforms can be built in a variety of ways. The two most common methods are use data lakes, which pull information into in a centralized hub to be accessed as necessary, and data mesh, which pulls together information from various sources through a knowledge graph.

Each method deals with centralization in different ways. Data lakes, for example, focus on the centralization of unstructured data, leveraging cloud services like Amazon Web Services and Google Cloud to store the information.

Data mesh takes an edge computing-style approach: It makes access to the data centralized. This is an appealing option for companies whose offices are spread across multiple locations. Last year, Sandipan Sarkar, a distinguished engineer of hybrid cloud transformation with IBM Global Business Services, told BizTech that data mesh can make information accessible no matter where it lives.

“As a product, data is discoverable through a metadata search and knowledge graph. The consumers — inside or outside the enterprise — may come and shop for the product that they are interested in,” he said. “They would not bother with how the data is prepared. They are simply focused on easily consuming it.”

The Role of Data Discovery in Modern Data Platforms

Data discovery is also essential to modern data platforms. This is a step beyond collecting and consuming data. During a data discovery process, information is analyzed for notable patterns with the help of artificial intelligence and machine learning. This discovery process can help you build an analytics strategy around raw data and identify digital transformation opportunities, according to Splunk’s Raihan.

“A modern data platform can be a true enabler for digital transformation, not only helping enterprises embrace new technologies for a better customer experience but also helping them create new revenue streams that weren’t possible before,” he says.

As organizations expand their use of AI, they also tend to expand their data-gathering efforts beyond just internal sources.

“We see so many companies as they finally get to that point of, ‘We’ve got great AI, great machine learning, and we’ve got all this internal data. But now, we also need external data,’” Catanzano says. “So, they start bringing in different data sets from lots of places, enriching the data they have. We see companies leveraging AI and ML across the modern data platform.”

LEARN MORE: How growing businesses can improve data storage performance.

Who Can Benefit from a Modern Data Platform?

One major benefit of the modern data platform is that it is designed with large and mid-sized enterprise in mind. By centralizing data or access to it, silos slowly dissolve and more coordinated efforts towards digital transformation can occur.

“What we’re seeing from a technology standpoint are a lot of great vendors driving that down to the point where, whether through Software as a Service models or otherwise, a small company can get the same benefits that large companies have gotten over the past couple of years,” Catanzano says. “That’s going to help accelerate this marketplace, because more people will be able to get to a whole modern data platform approach at a reasonable cost.”

If your organization is looking to take its digital transformation to the next level, data is a great place to start. Get acquainted with CDW’s data analytics solutions to learn what’s possible.