Álvaro Martín Lozano
Implementing continuous delivery in a data processing pipeline
#1about 4 minutes
From research concepts to production-ready data products
The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.
#2about 7 minutes
Core concepts of continuous delivery for data
Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.
#3about 11 minutes
Implementing a pipeline with immutable, versioned data
The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.
#4about 6 minutes
The challenge of orchestrating chained data jobs
Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.
#5about 5 minutes
Pros and cons of the immutable data approach
While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.
Related jobs
Jobs that call for the skills explored in this talk.
ROSEN Technology and Research Center GmbH
Osnabrück, Germany
Senior
TypeScript
React
+3
VECTOR Informatik
Stuttgart, Germany
Senior
Kubernetes
Terraform
+1
Matching moments
01:54 MIN
The growing importance of data and technology in HR
From Data Keeper to Culture Shaper: The Evolution of HR Across Growth Stages
01:53 MIN
The role of a freelancer integrated within a team
WeAreDevelopers LIVE – AI, Freelancing, Keeping Up with Tech and More
03:28 MIN
Why corporate AI adoption lags behind the hype
What 2025 Taught Us: A Year-End Special with Hung Lee
03:07 MIN
Final advice for developers adapting to AI
WeAreDevelopers LIVE – AI, Freelancing, Keeping Up with Tech and More
02:20 MIN
The evolving role of the machine learning engineer
AI in the Open and in Browsers - Tarek Ziadé
05:17 MIN
Shifting from traditional CVs to skill-based talent management
From Data Keeper to Culture Shaper: The Evolution of HR Across Growth Stages
09:10 MIN
How AI is changing the freelance developer experience
WeAreDevelopers LIVE – AI, Freelancing, Keeping Up with Tech and More
04:57 MIN
Increasing the value of talk recordings post-event
Cat Herding with Lions and Tigers - Christian Heilmann
Featured Partners
Related Videos
39:04Python-Based Data Streaming Pipelines Within Minutes
Bobur Umurzokov
30:46Charting the Journey to Continuous Deployment with a Value Stream Map
Josh Armitage
43:47Industrializing your Data Science capabilities
Dubravko Dolic & Hüdaverdi Cakir
29:36CI/CD Patterns and Antipatterns - Things your Pipeline Should (Not) Do
Daniel Raniz Raneland
44:40Enabling automated 1-click customer deployments with built-in quality and security
Christoph Ruggenthaler
27:52Progressive Delivery in Kubernetes
Carlos Sanchez
50:28Practical tips and tricks for CI/CD success
Zan Markan
29:25Modern Data Architectures need Software Engineering
Matthias Niehoff
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Neweasy
Bristol, United Kingdom
£182-195K
GIT
Azure
Spark
Kafka
+5
datasolut GmbH
Köln, Germany
Remote
€60-85K
Intermediate
Azure
DevOps
Terraform
+1
theHRchapter
Calp, Spain
Remote
DevOps
Agile Methodologies
Continuous Integration
Siemens AG
Berlin, Germany
C++
GIT
CMake
Linux
DevOps
+7
theHRchapter
Municipality of Valladolid, Spain
Remote
DevOps
Agile Methodologies
Continuous Integration
Novelus
Municipality of Madrid, Spain
Intermediate
Hadoop
PUEDATA
Municipality of Madrid, Spain
Remote
Intermediate
ETL
Java
Spark
Python