Tag: productized data platform
CI/CD for Data Workflows: Automating Prefect Deployments with GitHub Actions
The final part of the Data Platform Infrastructure on GCP series covers CI/CD for Prefect deployments using GitHub Actions and Docker. Automate flow builds, worker updates, and streamline orchestration across environments.
Getting to Your First Flow Run: Prefect Worker & Deployment Setup
Run your first data ingestion workflow with Prefect, Docker, and Kubernetes. This guide walks through containerized flow execution, Prefect worker deployment, and clean deployment configs, laying the foundation for a scalable, maintainable orchestration layer.
How to Setup Data Platform Infrastructure on Google Cloud Platform with Terraform
Learn how to set up a secure, scalable data platform infrastructure on Google Cloud Platform (GCP) using Terraform. This step-by-step guide covers VPC configuration, Compute Engine setup, firewall rules, Identity-Aware Proxy (IAP), Cloud NAT, and more, ensuring a cost-effective, flexible, and secure foundation for your data platform.
Deploying Prefect on any Cloud Using a Single Virtual Machine
A pattern to improve observability, monitoring and, ultimately, data operations with Prefect. We show how to find the right trade off between number of deployments and improved operations.
More tags:
- Access Management
- CI/CD
- Cloud Cost Management
- Data Engineering
- Data Governance
- Data Infrastructure
- Data Integration
- Data Platform
- Data Platform Optimization
- Dev vs Prod
- Entra ID
- Kubernetes
- Modern Data Stack
- Modularity
- Python
- Query Performance
- RBAC
- RFC Protocol
- SAP
- Security Architecture
- analytics workflow
- connectivity
- data ingestion
- data orchestration
- data pipelines
- data platform architecture
- data workflows
- dataops
- dlt
- elt
- elt process
- github actions
- google cloud platform
- governance
- networking
- orchestration
- pipelines
- prefect
- prefect worker
- productized data platform
- roles
- terraform
- workflow orchestration