2023 Vendor and Product Assessment
Data Pipelines
The development, testing and deployment of data pipelines is essential to generating intelligence from data. Just as a physical pipeline is used to transport water between turbines, generators and transformers in the generation of hydroelectric power, so data pipelines are used to transport data between the stages involved in data processing and analytics to generate business insight. Healthy data pipelines are necessary to ensure data is integrated and processed in the sequence required to generate business intelligence (BI).
The concept of the data pipeline is nothing new, but it is becoming increasingly important as organizations adapt data management processes to be more data driven. Data pipelines have traditionally involved batch extract, transform and load processes, but data-driven processes require more agile, continuous data processing as part of a DataOps approach to data management, with an increased focus on extract, load and transform (ELT) processes, as well as change data capture and automation and orchestration.