Blog

Running dbt Rescue Rebuild in Production: Operational Playbooks, Failure Models, and Recovery Patterns

2026-03-27T16:55:00Z

In a previous blog post, we introduced dbt_rerun, a dedicated Prefect deployment we use as a rescue mechanism for production dbt pipelines. We discussed the problem it solves, why standard recovery options fall short, how the deployment is wired, and where it fits into our day-to-day development workflow.

Now, let’s focus on what happens after you click “run”. We’ll walk through how we actually use dbt_rerun in production: choosing the right dependency scope, managing queue and warehouse contention, recovering from outages, handling incremental models safely, and validating that the rescue truly fixed the issue. These are the patterns and guardrails we rely on during real incidents, where speed matters but correctness matters more.

Outage Recovery

When an upstream failure or source outage takes down a large number of models at once, the rescue deployment becomes the primary recovery channel. Attempting to recover through the normal scheduled pipelines risks re-triggering failures before the source is stable, and provides less visibility into what has and hasn’t been rebuilt.

The recommended approach is to batch all affected models into a single, well-scoped rescue run:

dbt run --select int_model_1+ int_model_2+ +mart_model_1+ +mart_model_2+

This pattern has several advantages:

It is isolated from the main pipeline schedules, so a failure in the rescue run does not interfere with ongoing production work.
It runs the full lineage for each affected model, ensuring that intermediates and their dependent marts are rebuilt in the correct order.
It makes recovery explicit and auditable - we know exactly which models were confirmed rebuilt, rather than assuming the scheduler will handle it.

For very large outages (e.g., 50 or more staging models failing overnight), we group models by lineage affinity rather than triggering a single enormous run. This keeps individual runs fast, minimizes the blast radius of any secondary failure, and makes it easier to identify which chains have been recovered.

Resources Degradation

Warehouse resources are shared across all running pipelines. A rescue run that overlaps with a scheduled run that touches the same large models will introduce concurrency at the database level, leading to competing read and write operations on the same tables, which can cause both runs to slow significantly or time out.

Two situations are particularly risky:

Concurrent writes to the same table: If a rescue run and a scheduled run attempt to write to the same model at the same time, they will contend for table locks. The likely outcome is that both runs become slow or stall, requiring manual intervention.
Merging an updated model while it is actively running: When a Data Analyst merges a model change to main while a scheduled pipeline is mid-run and still executing that model, the downstream runs in that same pipeline may pick up the new code before the model has been rebuilt against it. This is a common source of unexpected failures. Our practice is to either merge after the current scheduled run cycle has completed or merge immediately after a cycle ends and trigger the rescue rebuild before the next cycle begins.

Warehouse resource contention is easiest to avoid by checking the Prefect run timeline before triggering a rescue: if a heavy scheduled deployment is running or about to run, we wait for it to clear.

Downstream and Upstream Reloads

Choosing the right dependency scope is one of the most consequential decisions in a rescue run. Running too narrow leaves downstream models stale; running too wide wastes resources and increases the risk of contention.

Here is a compact guide to the selection patterns we use:

+model - the model plus all upstream parents (everything it reads from)
model+ - the model plus all downstream children (everything that reads from it)
+model+ - both directions simultaneously

Intermediate models

Mart models

Using --exclude for partial progress recovery

When a large run times out mid-way, some models in the selection will have already been built. Re-running the full selection would rebuild them unnecessarily. The --exclude flag lets us resume from where we left off:

# Initial run: times out after building int_model, dep_1, dep_2 of 10 models
dbt run --select int_model+ --full-refresh

# Resume run: exclude the three already-built models
dbt run --select int_model+ --exclude int_model dep_1 dep_2 --full-refresh

This pattern is especially important for full-refresh runs on large incremental chains, where rebuilding from scratch would waste significant time and resources.

Incremental Models and Full-Refresh Strategy

Incremental models are where most rescue mistakes happen, because the wrong refresh strategy can silently lock in bad history or overload the warehouse.

Incremental models represent the most complex rescue scenario, because the decision of whether to use --full-refresh has lasting consequences for both data correctness and warehouse load. Our incremental models typically use a unique_key for deduplication and an is_incremental() filter to restrict which data is loaded on each run, often tied to a date cursor or a calendar condition, such as a monthly refresh window. Understanding both of those constraints is a prerequisite for choosing the right rescue approach.

When to use --full-refresh

Use --full-refresh when the change invalidates the historical state of the table:

The join logic, business rules, or transformations affecting existing rows have changed.
The unique_key definition has changed.
A bug produced an incorrect history that must be corrected.
A new column is being backfilled from source data that already exists in staging.

When not to use --full-refresh

Only future data will differ (for example, adding a new column with NULL values for historical rows is acceptable).
The model runs on a monthly cadence, and a normal incremental run will naturally pick up the change on its next scheduled execution.
The model is very large, and a targeted historical window backfill would be more efficient than a full rebuild.

When --full-refresh is genuinely necessary on a large incremental model, treat it as a scheduled operation: run it during a low-traffic window, ensure queues are free, and monitor warehouse resource usage throughout.

Write disposition and dbt materialization alignment

The choice of write disposition is tightly coupled to how the source generates data:

If source records are append-only and never modified, the incremental pattern with a date cursor is safe and efficient.
If source records can be updated, a unique_key merge strategy is required to avoid duplicate or stale rows in the target.
If correctness requires reconstructing the full table from scratch on a change, --full-refresh is the only reliable option, but it should be the last resort, not the default.

Cross-Model Dependencies: the Heavy Lineage Trap

Most rescue incidents do not involve a single model in isolation. They involve a model that sits in the middle of a deep lineage graph, with multiple upstream sources and multiple downstream consumers. When such a model is rescued carelessly, the blast radius extends well beyond its intended scope.

Before triggering a rescue run, we always inspect the lineage of the target model:

How many upstream models does it depend on, and how heavy are they?
How many downstream models reference it directly or transitively?
Are any of those downstream models incremental or particularly large?

If the lineage is deep and the models are heavy, we split the rescue into multiple runs, ordered by layer:

Rebuild the upstream models first to ensure stable inputs.
Rebuild the target model against the refreshed upstream.
Rebuild the downstream models once the target is confirmed correct.

This layered approach reduces the risk of a single timeout taking down the entire lineage recovery and makes it easier to identify failures at each stage.

Scheduling Heavy Reloads

For rescue operations that involve large incremental models, deep lineages, or full-refresh requirements, ad hoc triggering is rarely the right approach. Instead, we treat these as scheduled operations with the same care we apply to any production job.

Identifying a reload as “heavy” is the first step. A useful heuristic: if a model typically takes more than a few minutes in its normal incremental run, its full-refresh will be significantly longer - potentially by an order of magnitude. Dependency chains multiply this further.

Scheduling considerations:

Run heavy rescues outside peak hours, when scheduled pipelines have cleared their queues and warehouse load is low.
If multiple heavy models need to be rescued, stagger them across separate windows rather than running them in parallel, unless queues and warehouse headroom are confirmed sufficient.
Monitor the run actively - heavy full-refresh operations are the most common source of rescue-induced timeouts.

When a rescue run times out partway through, the --exclude pattern described in the downstream/upstream section allows the run to resume without duplicating already-completed work.

Recovery Practices

Consistent recovery practices reduce the operational costs of incidents and make rescue deployments easier to reason about.

Post-merge rebuild discipline. Every model merge should be followed immediately by a rescue rebuild, completed before the next scheduled run cycle begins. This eliminates the most common source of “new code, old table” mismatches. If there is insufficient time before the next cycle, the merge should wait for a better window.

Batching by lineage for outage recovery. When many models fail simultaneously, resist the temptation to address each one individually. Group them by shared lineage: models that share upstream sources or downstream consumers should be recovered together in a single selection string. This is faster, more reliable, and produces a cleaner audit trail.

Validating recovery before declaring success. A successful dbt run is necessary but not sufficient. After each rescue run, we verify:

dbt tests triggered by the deployment have passed.
Row counts and max dates in the rebuilt tables match expectations.
Downstream consumers are producing correct outputs in their next run.

Using the sandbox for validation before prod. For complex lineage changes, the sandbox schema - which mirrors the structure of all production schemas inside the production cluster - can be used to validate the rebuild logic before triggering it against production tables.

Pre-Run Checklist

Before triggering any rescue run, we work through the following:

Are SQL queues currently saturated, or is a heavy scheduled run about to start?
Is the target model heavy (large row volume, complex joins, or deep lineage)?
Have I mapped the upstream and downstream scope? Am I using the right selection (+model, model+, or +model+)?
Is the model incremental? If so, will a normal incremental run load any data today, or will the monthly filter make it a no-op?
Does the change require --full-refresh, or can a targeted backfill or normal incremental run achieve correctness?
Am I about to overlap with a scheduled run that touches the same lineage?
If the run times out, do I have a plan for resuming with --exclude?
After the run: did tests pass, and have I sanity-checked the output tables?

Summary

The dbt_rerun rescue deployment is one of the highest-leverage tools in our daily data service operations. It provides a controlled, production-safe channel for rebuilding changed or broken models without affecting the main pipeline schedules, which is essential for maintaining data trust in a production environment where models change continuously.

Used well, it functions as a scalpel: we rebuild the smallest correct scope, at the right time, with the right materialization strategy. Used carelessly ‒ triggering heavy full-refreshes at peak hours, overlapping with scheduled runs, or rebuilding too wide a lineage all at once ‒ it becomes a source of incidents rather than a remedy for them.

The practices described in this article ‒ queue awareness, lineage inspection, incremental strategy, and post-run validation ‒ reflect lessons learned from daily operational use. The principles, however, generalize: any on-demand dbt execution in a production environment benefits from the same discipline.

The Rescue dbt_rerun Deployment: Rebuilding Changed and Broken Models Without Disrupting Production

2026-03-23T13:45:00Z

Keeping production data correct after a model change is harder than it looks.

A Data Analyst merges a new dbt model (or a hotfix) to the main branch. The code is now live. But the table in the production database still reflects the old logic. Scheduled downstream runs will start referencing the new code within minutes, yet the upstream table hasn’t been rebuilt. The result is either a silent data mismatch or a cascade of failures across the pipeline.

That scenario plays out more often than it should, because the standard options for recovery are blunt instruments:

Waiting for the next scheduled run assumes the failure will resolve itself, which it often won’t.
Rerunning the entire pipeline is expensive, risky during peak hours, and almost always more than is needed.
Rerunning the wrong subset is fast but leaves downstream models in an inconsistent state.

A targeted, production-safe rescue mechanism solves all three problems at once.

In this article, we describe how our team uses a dedicated Prefect deployment ‒ dbt_rerun ‒ to handle exactly this: rebuilding changed, broken, or historically stale models with the right scope and at the right time, without disrupting the main pipeline schedules.

Our insights are grounded in daily operational experience running this deployment as a core part of our data service.

What We Are Solving For

Production dbt pipelines are rarely static. Models get updated, source schemas drift, upstream data arrives late or corrupt, and historical backfills are periodically requested. Each of these situations requires rebuilding some portion of the model graph ‒ but rebuilding the wrong scope, at the wrong time, or with the wrong materialization strategy can cause as much damage as the original failure.

The core challenge is surgical precision: rebuild exactly what needs to be rebuilt, in the right dependency order, without blocking scheduled runs or saturating warehouse resources.

Three recurring failure modes motivated us to formalize the rescue approach:

Merge without immediate rebuild. A Data Analyst merges an updated model into the main. Scheduled downstream runs start pulling from it before it’s been rebuilt, producing incorrect or failed outputs.
Outage recovery at scale. A weekend failure takes out dozens of intermediate models and their marts. Recovering each one individually is impractical; recovering everything blindly through the main pipeline is too risky.
Heavy full-refresh at peak hours. An incremental model is rebuilt with --full-refresh at rush hour, saturating SQL queues and degrading warehouse performance across unrelated pipelines.

The dbt_rerun deployment addresses all three by giving us a dedicated, controlled channel for on-demand model rebuilds.

The Solution: A Dedicated Rescue Deployment

Let’s start from the beginning: What do we mean by a “Rescue Deployment”?

By “rescue deployment", we mean an on-demand dbt execution path that is deliberately separated from scheduled production pipelines. The dbt_rerun deployment is not tied to a specific model, environment, or schedule. It exists solely to rebuild production tables when something has changed or broken, using an explicitly chosen dbt selection and timing. Unlike scheduled runs, it is triggered manually, scoped deliberately, and monitored closely from start to finish.

This separation is intentional: it allows us to fix production state without modifying schedules, redeploying code, or taking unnecessary risks during peak hours.

We run dbt_rerun as a Prefect deployment built on a shared template:

from templates import transform_and_catalog



transform_and_catalog(

    dbt_selects={"run": ""},  # Filled at runtime via Prefect custom run

    work_queue="dbt",         # Routes to the shared dbt work queue

)

The deployment is intentionally minimal. The dbt_selects["run"] parameter is left empty at definition time and filled in when a custom run is triggered in Prefect, allowing any valid dbt selection string to be passed without modifying code or creating new deployments. In most cases, this is simply the name of the model that needs to be rebuilt (e.g., int_orders or mart_revenue), though more complex selection strings can be used when the rescue scope requires it.

This means a single deployment covers the full range of rescue scenarios: post-merge rebuilds, hotfix recoveries, historical backfills, and outage remediation.

How the Deployment Is Wired

The transform_and_catalog template handles the orchestration plumbing. Here is an annotated version of the relevant configuration:

def transform_and_catalog(
    flow_branch: str | None = None,        # Branch where the Prefect flow code lives
    dbt_repo_branch: str = "main",       # dbt repo branch to run against
    schedules: list[str] = [],             # Optional cron schedules (empty for rescue)
    work_queue: str | None = None,         # Which Prefect queue executes this deployment
    version: int = 1,
    tags: list[str] = [],
    **kwargs,
) -> None:
    params = {
        "dbt_repo_url": "",
        "dbt_repo_token_secret": "",
        "dbt_repo_branch": dbt_repo_branch,
        "dbt_project_path": "dbt/lakehouse",
        "run_results_storage_path": "s3:///dbt/runs",
        "run_results_storage_credentials_secret": "",
    }
    params.update(**kwargs)


    deploy(
        name="",
        flow_name="transform_and_catalog",
        flow_branch=flow_branch,
        params=params,
        schedules=schedules,
        work_queue=work_queue,
        version=version,
        tags=tags,
    )

Two operational details are worth highlighting:

work_queue="dbt" routes the rescue run through the same queue pool as regular dbt deployments, making queue saturation visible and manageable.
dbt_selects passed at runtime means the deployment never needs to be redeployed to handle a new scenario. At trigger time, the engineer fills in the selection string in the Prefect UI custom run dialog.

The deployment also runs dbt’s built-in tests after each execution, so model correctness is validated automatically without a separate step.

Where Rescue Fits in Our Daily Workflow

Before diving into individual scenarios, here is how dbt_rerun fits into our standard development lifecycle:

Local development: A Data Analyst builds or updates a dbt model locally against the sandbox schema ‒ a schema inside the production cluster that mirrors the structure of all production schemas.
Pull request and merge: The Data Analyst opens a PR to main. Once approved and merged, the new or updated model code is live in the repository.
Rescue rebuild: We trigger **dbt_rerun** to rebuild the changed model along with the appropriate upstream and downstream dependencies. This is the step that makes the production table reflect the new code.
Scheduled runs proceed normally: Once the rebuild is confirmed, the regular pipeline schedules pick up cleanly without encountering a code/table mismatch.

This sequence applies equally to post-merge refreshes, hotfix recoveries, and outage remediations ‒ only the selection string and the urgency window change.

Lateness and Queue Contention

Triggering a rescue run at the wrong moment can cause more lateness than the original problem it is meant to solve.

Our production environment typically runs five SQL queues in parallel. When four of those are occupied by scheduled deployments, taking the fifth with a heavy rescue run means that any new scheduled run that needs a queue will be blocked until the rescue completes.

If the rescue model is large or triggers a long chain of dependencies, this blockage can cause data latency across multiple downstream consumers.

Before triggering any rescue run, we assess the following:

How many SQL queues are currently in use?
Is the target model light or heavy (in terms of compute and row volume)?
Does the lineage include other heavy models?
Are any scheduled runs likely to start in the next 15–30 minutes that would need a free queue?

If queues are near capacity and the model is heavy, we do not trigger immediately. Instead, we either wait for a quiet window or split the reload into smaller, sequential runs that each finish quickly and release the queue.

NOTE: A heavy --full-refresh on a large incremental model is particularly risky in this context. Not only does it occupy a queue for an extended period, but it also recreates the full table from scratch, generating significant warehouse load. We treat any full refresh of a large incremental model as a scheduled operation, not an ad hoc one.

The mitigation pattern for large reloads is to split by lineage: instead of one run covering ten dependent models, run three batches of three or four models each, in low-traffic windows with enough time between them for queue recovery.

What’s next

The dbt_rerun deployment gives us a safe, controlled way to correct production state when models change or break, without touching schedules or resorting to heavy-handed reruns. But the deployment itself is only part of the solution. How and when it is used matters just as much as how it is wired.

Watch out for part two of this post, where we’ll go deeper into the operational side: how we scope rescue runs in real incidents, how we recover from outages, how we handle incremental models and full-refresh decisions, and how we avoid turning a rescue into a second production issue. If you’re the one on call when data breaks, stay tuned.

Why Data Teams Struggle Without Separate Dev and Prod Environments

2026-01-22T13:00:00Z

It’s Monday morning. The CEO’s dashboard shows zeros. Sales metrics are gone. The data team is digging through logs, trying to figure out which Friday deployment broke production. Familiar scenario?

Unfortunately, this isn’t just an anecdote. Over the past three years, 50% of data centers experienced at least one impactful outage. Of these incidents, nearly 40% were caused by human error, with 85% stemming from staff failing to follow procedures or from process flaws ‒ issues that could often be avoided if analytics and production workloads were properly separated. And the consequences can be severe: more than half of these outages cost organizations over $100,000, and one in five exceeded $1 million.

This happens when teams share the same data warehouse, the same jobs, and sometimes even the same credentials. At first, it feels efficient: no duplicate infrastructure, no setup overhead. But over time, it creates a fragile system in which even small changes can ripple into business-critical outages.

When Development and Production Collide

When both environments are the same, every update carries high stakes. Test directly in production, and you risk breaking a dashboard the CFO uses daily. Hold back changes, and you slow down every initiative.

I’ve seen this play out in many companies:

Stakeholders stop trusting reports because they fail randomly ‒ and this is not just my experience: about three-quarters of organizations say business stakeholders are the ones who spot issues first, most of the time.
Business users build their own spreadsheets and one-off tools to “fix” gaps.
Data teams spend their time firefighting instead of improving pipelines. On average, data engineers spend 40% of their time (roughly 2 days per week!) addressing bad data and unplanned issues.
Deployment windows shrink to late nights and weekends because no one trusts changes during business hours. And it’s not just time lost: over 70% of technology staff report being negatively impacted by unplanned work in three or more ways, including heightened stress and anxiety, reduced work-life balance, and less time to focus on strategic projects.

For the people working inside these environments, the stress is constant. Engineers avoid trying new ideas because the cost of failure is too high. The pace of delivery slows, and eventually, good people leave for organizations where they can focus on building rather than patching.

What a Healthy Setup Looks Like

The solution is not complicated in principle: give development and production their own space. That means separate cloud accounts, databases, compute resources, and access controls. Tricks like using schema prefixes in the same warehouse don’t solve the problem ‒ they only create false security.

A good development environment doesn’t need to be a full copy of production. It just needs to behave the same way. I live by those 3 golden practices for achieving this:

Use infrastructure-as-code to keep environments consistent.
Create smaller datasets that are representative of production (with sensitive data masked).
Set up a clear path for code to move: dev → staging → production, with tests and reviews at each step.

Version control (Git) underpins all of this. Every change should leave a trail, so you can review, roll back, and understand what’s running where.

Rolling It Out Without Overwhelming Teams

A shift like this doesn’t happen overnight, and it shouldn’t. Most teams succeed by taking it step by step:

Set up separate infrastructure ‒ provision isolated development resources using infrastructure-as-code so environments stay consistent and secure. Check our blog post on how to do it using Terraform, a great starting point for provisioning dev infrastructure.
Get usable data into dev – establish pipelines to copy and mask subsets of production data. You can see some best practices in our piece on building ingestion pipelines with dlt and Prefect.
Define workflows and train the team – document how changes flow, how reviews are conducted, and the promotion criteria.
Automate deployments – CI/CD pipelines handle testing, validation, and approval gates before changes reach production.
Add monitoring – make sure each environment has the right level of alerting so issues are caught quickly. Head to our Prefect deployment guide for a practical approach to monitoring and automation, which also covers the previous step on deployment.

The Payoff: Stability, Speed, and Trust

Companies that make this switch often see significant drops in incident rates within the first six months ‒ based on our experience 50% in 3 months. Deployments that once happened monthly shift to a weekly or even daily rhythm. Teams finally get space to experiment without the fear of breaking production, and business leaders start trusting dashboards again because they consistently work.

Inside the team, the shift is just as important. Instead of late-night firefighting, data professionals can focus on delivering value, exploring new tools, and building solutions that last.

“But We’re Different…”

Yes, yes, I know. “It’s too expensive, too complicated, or not suitable for the amount of data we handle.” But here’s the reality:

"We can’t afford separate environments.”

Cloud platforms make it manageable. You don’t need full-scale production replicas; lightweight development instances are enough. For example, on Azure Synapse Analytics, small teams with 1TB of data and 5–10 users typically spend $1,500–4,000 per month per environment using serverless SQL pools with pause/resume capabilities. On AWS Redshift Serverless, similar workloads run $1,000–3,000 per month since you only pay for actual query time—no idle clusters burning money overnight. For even lighter workloads, Amazon Athena can drop costs to $200–800 per month when you’re querying well-partitioned data in S3 (at $5 per TB scanned). Compare that cost to your last outage.

“We’re too small for such complexity.”

Small teams need it most. If three people spend about 40% their time fixing broken production jobs, that effectively means more than one full team member’s capacity is devoted to patching problems instead of innovating or building new solutions. That’s a huge hit to both productivity and morale.

“Our data is too big to duplicate.”

You don’t need to. Subsets, samples, or synthetic data usually give you what you need to test logic and performance.

Taking the First Step

Separating dev and prod isn’t an optional best practice. It’s the foundation for stable, trustworthy analytics. You don’t need to implement everything in one go, but starting small pays off quickly.

Think of it less as technical overhead and more as business enablement. When your analytics are reliable, your decisions improve. When deployments are faster, your team can respond to new business needs. When incidents are fewer, you unlock time for innovation.

The question isn’t whether your organization needs this ‒ it’s whether you want to start proactively or wait until the next production incident forces the change.

Data Platform Cost Optimization: Practical Strategies for Query Performance, Storage, and Cloud Resource Management

2025-10-27T13:06:00Z

Data platform costs can quickly spiral out of control, often catching organizations off guard as workloads scale and data volumes increase. Whether you’re a data engineer optimizing transformation pipelines, a data analyst writing complex queries, or a DevOps engineer managing infrastructure across AWS, GCP, and Azure, you have direct influence over your platform’s total cost of ownership.

Cost optimization isn’t the sole responsibility of a single team or role. Every decision made throughout the data platform lifecycle affects the bottom line. A poorly written query can consume hundreds of dollars in compute resources within minutes. Unnecessary data retention policies can bloat storage costs month after month. Inadequate resource planning can lead to over-provisioned infrastructure that runs idle most of the time.

This guide looks at cost optimization from multiple angles, recognizing that effective cost management requires collaboration across the entire data platform team. You’ll find practical strategies tailored to different aspects of data platform operations, from warehouse query optimization and incremental loading techniques to cloud resource reservations and data access controls. Each section highlights key principles and links to implementation approaches for deeper technical coverage.

The goal is simple: to help you identify and act on cost-saving opportunities in your area of expertise, building data platforms that are both efficient and economical.

Query Optimization for Lower Data Warehouse Costs

Query optimization is one of the most impactful ways to reduce platform expenses. A well-tuned query can run 10–100 times faster than an unoptimized one, directly lowering compute costs and improving user experience. Before writing SQL, how tables are structured, data types chosen, and storage organized determines achievable performance ceilings. Messy source tables with unnecessary columns, unoptimized types (eg. STRING instead of INT64 for IDs), and poor partitioning or clustering make queries orders of magnitude slower—even with well-written SQL. Universal architectural principles apply across all modern warehouses: partitioning, clustering/sorting, indexing, compression, and columnar storage. Start optimization here before platform-specific tuning.

Warehouse-Specific Techniques:

Each data warehouse behaves differently. BigQuery charges based on bytes processed or slot usage, making partition pruning and clustering critical for cost control. In Redshift, performance depends heavily on choosing the right distribution and sort keys to reduce data movement during joins. Understanding these platform-specific optimizations allows you to reduce query costs dramatically without changing business logic.

Execution Plan Analysis:

Understanding query execution plans is essential for identifying performance bottlenecks. Analyze the most expensive queries using BigQuery’s INFORMATION_SCHEMA.JOBS or Redshift’s SYS_QUERY_HISTORY (for serverless) and SVL_QUERY_REPORT (for provisioned clusters). Focus optimization on queries consuming the most slots, scan time, or compute resources.

Precomputation and Caching:

Choose the right materialization strategy for recurring queries. Physical tables offer the most flexibility for complex transformation logic and refresh schedules but require explicit pipeline management. Standard views provide real-time data access without storage overhead, ideal when freshness is critical and query performance is acceptable. Materialized views physically store precomputed results for faster reads but come with platform-specific limitations around refresh behavior, join complexity, and aggregation support—evaluate whether your use case fits before committing.

Complement materialization with result caching: BigQuery caches identical query results for 24 hours at no charge, while Redshift keeps result sets in memory. Both eliminate redundant computation when users re-run the same queries.

Optimizing Joins and Complexity:

Simplify query logic by breaking down complex queries into intermediate tables. Filter data before joining to minimize scanned bytes and reduce overall processing costs. Make sure to keep the larger table on the left side of the join.

Advanced Data Loading Techniques:

Efficient data loading practices can lower compute and storage costs while improving pipeline performance. Instead of reprocessing entire datasets, incremental loading focuses only on what has changed, cutting ingestion costs by up to 90%.

Incremental Loading Patterns:

The high-watermark method is a simple and reliable approach: track the maximum timestamp or sequential ID from the previous load, then query for records exceeding that watermark. For full change tracking, including deletes, implement Change Data Capture (CDC) using tools like dltHub, a Python-based library that supports CDC and incremental loading out of the box. It handles schema evolution, state tracking, and normalization automatically.

Batch vs. Streaming Trade-offs:

Batch loading offers significant cost advantages for most analytical workloads, with traditional batch jobs running on scheduled intervals (hourly, daily) at minimal infrastructure cost. Event-based batch processing provides a middle ground, triggering data loads when specific events occur, such as new files arriving in object storage or source system notifications, eliminating unnecessary empty runs while maintaining batch processing efficiency. Micro-batch processing (near real-time) has emerged as a viable option for many use cases, processing small batches of data every 5-10 minutes and satisfying business requirements that don’t truly need sub-minute latency. True streaming ingestion suits scenarios requiring sub-second latency but comes at premium pricing due to continuous resource consumption and the need for separate, specialized data platform components. This is often confusing for less technical staff, but if you realize that a 1 second frequency is 300 times more frequent than 5 minutes, it should make intuitive sense that a different approach is required. Evaluate whether business requirements genuinely need real-time data or if event-triggered or micro-batch processing satisfies use cases at a fraction of the cost—many “real-time” dashboards function perfectly well with 5-15 minute refresh intervals while reducing infrastructure expenses significantly.

Optimal Scheduling:

When scheduling batch jobs (instead of event-based processing), run them during off-peak hours to reduce slot contention and improve query performance. "Distribute pipeline start times based on data dependencies and SLAs to avoid unnecessary resource spikes.

Cloud Resource Management and Cost Control

Infrastructure typically represents one of the largest expenses in a data platform, yet many organizations run their warehouses at default configurations. Smart resource management requires understanding workload patterns and matching the right pricing model to each use case.

Reserved Capacity Strategy:

Where possible, commit to reserved capacity for predictable, steady-state workloads. BigQuery with committed slot reservations can save 40-60% compared to on-demand for high-usage scenarios (docs). Redshift Reserved Instances provide up to 62.5% savings with 1-year or 3-year commitments. Analyze your past 3-6 months of usage to identify the minimum baseline capacity required during low-demand periods - this becomes your reserved capacity floor. Configure auto-scaling on top of this baseline to handle variable workloads and peak demand, ensuring you only pay for additional resources when actually needed rather than over-provisioning for worst-case scenarios. This hybrid approach balances cost predictability through reservations with elasticity through auto-scaling, optimizing both budget and performance. Below you can find an example pricing discount for Redshift Reserved Instances:

Auto-Scaling Configuration:

Configure auto-scaling for workloads outside your core data warehouse. For transformation engines and orchestration runners, implement horizontal scaling based on queue depth or CPU utilization. Set aggressive scale-down policies during off-hours when data pipelines experience minimal activity, automatically capturing savings rather than running fixed capacity 24/7.

Spot Instance Usage:

Use spot instances (AWS) or preemptible VMs (GCP) for fault-tolerant batch workloads at 60-90% discounts. Historical data backfills, large-scale data quality checks, and experimental analytics workloads run well on spot capacity. Implement retry logic and design jobs to checkpoint progress periodically, avoiding spot instances only for real-time pipelines or time-sensitive reporting.

Data Access Control Strategy

The way users access data significantly impacts platform costs. Centralized architectures push all queries through a single warehouse, creating bottlenecks and driving up compute usage.

Self-Service vs. Centralized Reporting:

Self-service analytics tools empower users but can lead to inefficient ad-hoc queries that consume excessive resources. Balance this by providing pre-aggregated datasets, semantic layers, and curated data marts for common analysis patterns and proper training. Centralized reporting through scheduled dashboards and reports consolidates repetitive queries into single executions shared across users.

Aggregation Layers:

Create data summarization and aggregation layers that pre-compute common metrics at various granularities. Daily, weekly, and monthly aggregation tables reduce the need to scan massive fact tables repeatedly. Users query these optimized datasets instead of raw transaction data, dramatically reducing bytes processed and query execution times.

Access Control Patterns:

Implement role-based access control in order to limit expensive query executions. Grant read-only access to pre-aggregated views for most analysts while restricting raw table access to data engineers who understand optimization techniques. Configure query timeout limits and result set size restrictions to prevent runaway queries from consuming excessive resources.

Storage Optimization and Data Lifecycle Management

Storage often accounts for 30-40% of total data platform costs, yet receives less attention than compute optimization. Smart storage strategies can deliver major savings with minimal operational impact.

Data Lifecycle Management:

Automate data tiering into hot/warm/cold storage based on data age and access patterns. BigQuery offers long-term storage pricing (50% discount) for tables not edited in 90 days. Redshift supports automatic table optimization that moves cold data to S3. Design lifecycle policies around actual business needs instead of keeping everything in expensive hot storage.

Compression and Encoding:

Apply appropriate compression and encoding techniques for your data types. Redshift offers multiple encoding options (LZO, ZSTD, Byte-dictionary) that can reduce storage by 70-90% while improving query performance through reduced I/O. BigQuery automatically compresses data, but choosing appropriate data types (INT64 vs. STRING for numeric IDs) impacts compressed size significantly.

Time Travel for Point-in-Time Analysis:

Replace expensive SCD Type 2 dimension tables with native time travel capabilities where available. Traditional SCD Type 2 implementations duplicate dimension records for every change, creating surrogate keys, effective dates, and current flags that bloat storage and slow query performance as tables grow. BigQuery’s time travel feature allows querying data as it existed up to 7 days ago without maintaining separate historical versions, while extended time travel (up to 90 days) is available through snapshot configuration. Data lakehouse formats like Delta Lake, Apache Iceberg, and Apache Hudi provide similar capabilities through native versioning, enabling point-in-time queries without the complexity and storage costs of maintaining full SCD pipelines. For dimensions with frequent changes, time travel can reduce storage by 60-80% compared to SCD Type 2 while simplifying ETL logic and improving join performance by eliminating multi-version dimension lookups. Evaluate time travel as the default pattern for historical analysis, reserving SCD Type 2 only for regulatory requirements demanding explicit audit trails beyond platform retention windows.

Conclusion

Cost optimization for data platforms requires a balanced approach across query performance, data loading, infrastructure, access, and storage. No single change solves everything, but applying several targeted strategies can reduce expenses by 40–60% while maintaining or even improving performance.

Start by identifying your biggest cost drivers through monitoring tools, then focus on the highest-impact areas first. Quick wins like caching, partitioning, and incremental loading deliver immediate savings, while longer-term efforts like lifecycle management and data mesh adoption provide sustainable efficiency.

By applying these techniques consistently, data teams can build platforms that scale intelligently, delivering both performance and value.

SAP Data Ingestion with Python: A Technical Breakdown of Using the SAP RFC Protocol

2025-09-15T08:00:00Z

If you’ve ever tried to get data out of SAP, you know it’s often easier said than done. Manual exports take up a lot of time, standard tools don’t always fit real needs, and moving data into modern analytics or machine learning pipelines can feel like forcing two worlds together.

For years, many teams relied on pyRFC, the official Python library for SAP Remote Function Calls. But with pyRFC being decommissioned, there’s now a real gap for anyone who wants to keep integrating SAP data into Python workflows.

That’s why we’re working on a new Python library to replace pyRFC, focused on making SAP data ingestion easier, more reliable, and better suited for modern data workflows. A connector like this can directly call SAP functions, pull large tables in manageable chunks, and plug that data into Python pipelines. For data engineers, analysts, and developers, this isn’t about high-end tech for its own sake; it’s about saving time, reducing errors, and finally being able to use SAP data in the tools we already work with every day.

To dig into how this actually works, I’ve talked with Dominik ‒ a senior software engineer with more than ten years of experience, a computer science lecturer, and the tech lead of this project. In this conversation, he shares his knowledge, best practices, and lessons learned while building an SAP connector in Python using the RFC protocol.

______

Can you give us a general overview of how SAP RFC works?

SAP Remote Function Call is a communication mechanism that allows one system to execute functions in another system as if they were local calls. It is primarily used to integrate different SAP modules, connect SAP with external applications, and support distributed processing. RFC works by exposing specific function modules in SAP that are marked as “remote-enabled,” meaning they can be invoked across system boundaries. When an RFC is triggered, the caller system packages the request, sends it over the network, and waits for a response (in synchronous mode) or continues processing without waiting (in asynchronous mode). This approach provides a standardized and reliable way for SAP systems and external programs to exchange data and trigger business logic, making RFC a cornerstone of SAP interoperability.

What are some of the challenges of using SAP RFC to ingest data with Python?

One of the main challenges of using SAP RFC for ingesting data with Python is handling the very large data volumes that SAP tables can produce. RFC itself has technical limitations, such as row size restrictions such as:

Row size limitations
- The 512-character per-row restriction in RFC_READ_TABLE means wide tables with many columns need to be split across multiple queries.
- Reconstructing the full row in Python requires careful mapping of column segments.
Volume and performance
- Extracting millions of rows can be slow and may degrade SAP system performance if not throttled.
- Network latency and RFC protocol overhead can become bottlenecks for very large datasets.
- Lack of streaming support means all results are buffered in memory, risking high RAM usage in Python.
Pagination and batching
- Since RFC has no built-in pagination, developers need to implement logic to fetch data in smaller chunks.
- Requires careful handling of row offsets and consistency to avoid duplicates or missing records.
Data type handling
- SAP tables contain many proprietary data types (e.g., RAW, DEC, DATS, TIMS) that need explicit conversion to Python types.
- Inconsistent formatting (e.g., leading zeros, fixed-length fields) can require custom parsing logic.
Error handling and robustness
- Large queries can lead to timeouts or aborted RFC sessions.
- Error messages from SAP may be cryptic and require domain knowledge to interpret.
- Retry logic and fault tolerance are not built in and must be handled in the Python layer.
Security and access restrictions
- Not all tables are directly accessible due to SAP authorization profiles.
- RFC users often have limited permissions, which may block some required data.
Alternative interfaces
- RFC_READ_TABLE is convenient but not officially intended for large-scale data extraction.
- In some cases, more efficient solutions (e.g., SAP OData services, CDS views, or custom ABAP reports) may be required instead of RFC.
Connection/session handling
- With Python’s pyrfc, each call often opens a new RFC session, and connections can drop unexpectedly if not managed carefully. This leads to overhead in session initialization and can cause instability in long-running jobs.
- A custom C++ connector can maintain a persistent session without frequent disconnects, providing more stability and efficiency for large-scale or continuous data ingestion.

This means developers must implement batching, efficient memory handling, and sometimes parallelization to make ingestion practical. Without these techniques, performance can degrade quickly, and data may be truncated or lost during transfer, making large-scale SAP-to-Python integration non-trivial.

Can you explain how you interface a C++ library with Python?

Interfacing a C++ library with Python involves creating a thin wrapper layer that makes the C++ functions and classes look like native Python objects. To achieve that, we use **pybind11, **which handles this translation by generating a Python extension module that directly calls into the compiled C++ code. Once built, the module can be imported into Python just like any other package, allowing Python code to invoke high-performance C++ logic seamlessly. This approach avoids costly inter-process communication and provides a clean way to combine Python’s flexibility with the speed and efficiency of C++.

How can ingestion speed be optimized?

Ingestion speed from SAP via RFC can be optimized by designing smart calls that minimize the number of round-trip calls to the SAP system. Instead of pulling entire tables blindly, it’s often more efficient to filter data at the source, select only the required columns, and batch large queries into manageable but sizable chunks. This reduces overhead and avoids overwhelming the Python client with too many small calls. Another common strategy is to push down as much logic as possible into SAP—using where clauses, ranges, or custom remote-enabled function modules—so that Python only receives the data that is truly needed. By shrinking the number of SAP calls in this way, the integration pipeline becomes more efficient, faster, and less resource-intensive on both ends.

Why do you consider pyRFC “slow” when it comes to data ingestion?

pyRFC is not inherently “slow” for small to medium RFC calls, but it becomes inefficient when used for large-scale data ingestion from SAP tables. Several factors contribute to this:

Session management overhead
- pyRFC often opens and tears down RFC sessions per call, rather than maintaining a long-lived persistent session. This adds noticeable latency when thousands of calls are required for wide or paginated tables.
Row size and query splitting
- Because of the 512-character row size limitation in RFC_READ_TABLE, wide tables must be split into multiple queries. In pyRFC, reconstructing results requires extra calls and significant Python-side processing, slowing ingestion.
Serialization and conversion costs
- SAP data types (e.g., packed decimals, dates, times, raw fields) must be converted into Python objects. This conversion layer, implemented in Python, adds overhead compared to a native C++ implementation.
Global Interpreter Lock (GIL)
- Python’s GIL prevents true multithreaded parallel RFC calls within the same process. This limits scalability for high-throughput extraction workloads unless you resort to multiprocessing (which adds its own overhead). We are trying to jump over this problem using C++ library.
Error recovery and retries
- pyRFC connections may drop unexpectedly under load, requiring reconnections and retries. This increases latency compared to a C++ connector that maintains stable sessions.

In contrast, a native C++ RFC client avoids much of this overhead by:

Keeping a persistent connection/session.
Handling large data more efficiently with lower-level memory management.
Offering faster type conversions without Python’s object allocation overhead.

Can incremental ingestions be done using SAP RFC? If yes, how?

Incremental ingestions with SAP RFC are not straightforward, because RFC itself is just a transport mechanism and doesn’t provide built-in change tracking. In most cases, you cannot simply ask RFC for “only new or updated records” unless the underlying SAP function module or table has fields that can support this, such as timestamps or change indicators. If those exist, you can design your RFC queries in Python to fetch only rows newer than the last ingestion run, effectively simulating incremental loading. Otherwise, true incremental ingestion is better handled through SAP’s dedicated frameworks like ODP (Operational Data Provisioning) or CDS views, which are specifically designed for delta handling. So while basic RFC on its own doesn’t guarantee incremental ingestions, with careful design and the right SAP data sources, it can sometimes be approximated.

Can you give us a couple of code examples in Python on ingesting the data?

Connection:

def con(self) -> sap_rfc_connector.SapRfcConnector:
    """The C+++ connection to SAP."""
    if self._con is not None:
        return self._con
    con = sap_rfc_connector.SapRfcConnector(**self.credentials)
    self._con = con
    return con

Call:

def call(self, func: str, *args, **kwargs) -> dict[str, Any]:
    """Call a SAP RFC function."""
    func_caller = sap_rfc_connector.SapFunctionCaller(self.con)   
    result = func_caller.smart_call(func, *args, **kwargs)

    return result

One of the approaches to avoid pandas for the data ingestion (POC):

# Check and skip if there is no data returned.
try:
    if response["DATA"]:
        logger.debug("checking data")
        if print_regular:
            print_regular("checking data")
        record_key = "WA"
        data_raw = np.array(response["DATA"])
        
        # Save raw data to CSV file immediately
        logger.debug(f"Saving {len(data_raw)} rows to CSV file...")
        if print_regular:
            print_regular(f"Saving {len(data_raw)} rows to CSV file...")
        with open(temp_csv_path, 'w', newline='', encoding='utf-8') as csvfile:
            writer = csv.writer(csvfile)
            # Write header
            writer.writerow(fields)
            # Write data rows
            for row_data in data_raw:
                try:
                    split_data = row_data[record_key].split(sep)
                    if len(split_data) == len(fields):
                        writer.writerow(split_data)
                    else:
                        logger.warning(f"Row data length mismatch: expected {len(fields)}, got {len(split_data)}")
                except Exception as e:
                    logger.error(f"Error processing row: {e}")
                    continue
        
        logger.debug(f"Saved data to {temp_csv_path}")
        if print_regular:
            print_regular(f"Saved data to {temp_csv_path}")
        del response
        del data_raw
except Exception as e:
    logger.error(f"Error: {e}")
    if print_regular:
        print_regular(f"Error: {e}")
    break

_________

Conclusion

Dominik’s perspective makes it clear that working with SAP data through RFC is full of opportunities, but far from straightforward. From row size limits and type conversions to performance tuning and incremental loading, every step has its own challenges. His approach ‒ combining Python with a C++ connector, careful batching, and smart query design ‒ shows what it takes to make ingestion both reliable and efficient.

With pyRFC being decommissioned, these insights feel especially timely. They point to what’s needed in the next generation of connectors: tools that handle scale gracefully, integrate naturally into Python workflows, and make SAP data easier to work with daily.

CI/CD for Data Workflows: Automating Prefect Deployments with GitHub Actions

2025-07-03T11:00:00Z

You’ve built a Prefect flow that runs, but wondering what’s next?

If deploying it means copying files, running CLI commands, or manually registering deployments, you’re doing too much. In this guide, we’ll walk through how to automate Prefect deployments using GitHub Actions and Docker, so your flows move from dev to prod with zero manual steps. Cleaner workflows, fewer errors, and no more “did I forget to deploy that?" moments.

Welcome to Part 4 of our data platform series, where we bring automation and resilience to the forefront by introducing CI/CD for your data workflows. If you’ve followed along, you’ve seen how each layer builds on the last: from architecture through infrastructure to operational readiness. But just to be sure we’re on the same page…

Let’s recap…

In Part 1 (Deploying Prefect on any Cloud Using a Single Virtual Machine), we explored the architectural foundations of a modern data platform. We discussed why simplicity, flexibility, and scalability matter, and how a lightweight Kubernetes setup on a single VM can deliver immediate value while laying the groundwork for future growth.

Part 2 (How to Setup Data Platform Infrastructure on Google Cloud Platform with Terraform) moved us from theory to practice, automating cloud infrastructure using Terraform. It emphasized cloud-agnostic design while preserving the architectural principles from the first article.

We operationalized the platform in Part 3 (Getting to Your First Flow Run: Prefect Worker and Deployment Setup). You learned how to build a containerized execution environment, configure Prefect workers, and organize deployment code, culminating in your first successful data ingestion flow.

Now, in Part 4, we turn our attention to automation. You’ll learn how to implement a robust CI/CD pipeline using GitHub Actions, tailored for data platforms. We’ll break down three essential types of workflows (covering container management, infrastructure updates, and workflow orchestration) that form a scalable, resilient, and low-maintenance deployment system together.

Whether your goal is to reduce manual intervention, boost reliability, or accelerate delivery, this final part will equip you with patterns and real-world guidance to make your data platform production-ready. Let’s delve in!

CI/CD Foundations for Data Platforms

Why is CI/CD so important?

CI/CD (Continuous Integration and Continuous Deployment) is key to building reliable, scalable, and collaborative data platforms. Manual processes, such as registering deployments by hand or managing configurations outside of version control, can easily slow things down and lead to mistakes. Here’s how automation makes a big difference in modern data workflows:

Manual deployment steps are easy to mess up.

When people manually update deployments or configurations, there’s a higher chance of errors, inconsistencies, or missed steps. Automation ensures each deployment follows the same steps every time, cutting down on mistakes and keeping things running smoothly.

Without a shared codebase, it’s hard to stay aligned.

If local development isn’t tied closely to a shared, version-controlled code repository, keeping track of changes, rolling back mistakes, or working well as a team is tough. CI/CD pipelines enforce the use of a central repository, making the entire platform transparent and auditable.

Testing only locally hides problems.

Without automated workflows to provision and test in staging or development environments, teams often default to running tests locally. This limits visibility into how changes will behave in real-world conditions and increases the risk of production issues.

Manual processes don’t scale well.

As your data platform and team grow, manual processes quickly become bottlenecks. Automated CI/CD pipelines make it easier to bring on new team members, keep deployments consistent, and move faster, without losing quality or stability.

Setting up a strong CI/CD foundation creates a more stable, transparent, and scalable environment. It lets your data engineers spend more time delivering value and less time fixing avoidable problems.

What makes data pipeline CI/CD different from traditional software?

CI/CD for data platforms comes with its own set of challenges, especially compared to traditional software development. This is mainly because tools like Prefect act as orchestrators, running workflows inside separate, containerized environments. This setup introduces a few extra layers to manage:

Different Workflows for Different Parts

Docker Image Pipeline: Dedicated workflow for building, testing, and deploying container images, including all the flow dependencies.
Prefect Worker Management: The only long-living process requiring separate CI/CD for updates of the application itself and the base job template used in flow runs.
Deployment Configuration: Independent workflow for managing Prefect deployment definitions and versioning.

More Moving Parts to Coordinate

Each part of the system might have its own release schedule, so updates must be carefully timed to avoid breaking things.
The container environments used for development, testing, and production must match, or you risk inconsistencies and surprises.

Unlike in traditional software, where a single build moves through environments, data platforms rely on multiple components that each need to be managed separately. Because of this, automation has to be thoughtfully designed to keep everything in sync. Done right, it helps teams move faster without sacrificing reliability.

GitHub CI/CD Workflows for a Data Platform

Repository structure

Before diving into the workflows, here’s a quick look at the repository structure established so far. While this isn’t the complete repository, the following directories and files are the main ones that trigger CI/CD processes:

📦 repository
 ┣ 📂 .github
 ┃ ┣ 📂 workflows
 ┃ ┃ ┣ 📜 workflow-x.yml
 ┃ ┃ ┣ 📜 template-x.yml
 ┣ 📂 etc
 ┃ ┣ 📂 docker
 ┃ ┃ ┣ 📜 Dockerfile
 ┃ ┣ 📂 helm_values
 ┃ ┣ ┣ 📂 prefect-worker
 ┃ ┃ ┃ ┣ 📜 values-dev.yaml
 ┃ ┃ ┃ ┣ 📜 values-prod.yaml
 ┣ 📂 src
 ┃ ┣ 📂 edp_flows
 ┃ ┣ ┣ 📂 flows
 ┃ ┃ ┃ ┣ 📜 flow-x.yml
 ┣ 📜 pyproject.toml
 ┗ 📜 prefect.yml

CI/CD Workflows overview

With the structure in place, let’s walk through the three main workflows that drive CI/CD for this data platform. This setup is designed to be portable and not tied specifically to GitHub Actions, and its goal is to automate flow deployment with Prefect, avoid unnecessary Docker builds, and use Prefect’s GitHub Repository Block to manage flows cleanly. The three key workflows are:

Flows Image Workflow: Triggered when updates to flow image dependencies are needed.
Prefect Worker Workflow: Runs when changes are made to the base job template or worker configuration.
Prefect Deployments CI/CD: Used for developing and managing new deployments and flows; this is the main workflow during development.

Let’s take a closer look at all of them.

Workflow 1: Flows Image Builder

This workflow is triggered by changes to any of the following files:

etc/docker/Dockerfile
pyproject.toml

Pull request steps

Check version increase: Validates that the version number was bumped correctly (e.g., 1.2.3 → 1.2.4, 1.3.0, or 2.0.0). To calculate acceptable versions after an increase, there is the christian-draeger/increment-semantic-version@1.2.3 action for GitHub that can calculate the next patch, minor, and major version, which can later be compared with the actual version that was manually increased by the developer.
Build DEV Image: Builds a unique, versioned DEV image for the edp-flows. It can be tagged using the pattern:

${VERSION}-pr-${{ github.event.number }}-run-${{ github.run_number }}

The image is then pushed to the GitHub Container Registry. In the previous blog post, we prepared an example Dockerfile. Here’s what a GitHub workflow to handle it might look like:

jobs:
  build-and-push:
    name: Build and push docker image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Create a multi-platform builder
        run: |
          docker buildx create --name builder --driver docker-container --use
          docker buildx inspect --bootstrap

      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ inputs.registry }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: "Build and Push Docker Image"
        uses: docker/build-push-action@v6
        with:
          context: ./
          file: ${{ inputs.dockerfile }}
          platforms: ${{ inputs.platforms }}
          push: ${{ inputs.push }}
          tags: ${{ inputs.registry }}/${{ inputs.organization}}/${{ inputs.image_name }}:${{ inputs.image_tag }}

      - name: Cleanup the builder
        run: docker buildx rm builder

Update DEV Prefect work pool: Updates the base job template on the DEV environment to reference the newly built Docker image. During this step, it is essential to update the image inside the baseJobTemplate definition to the newly created one, which can be handled even with a simple replacement:

jobs:
  prefect-worker-helm:
    name: Prepare prefect worker
    runs-on: ${{ inputs.runs_on }}
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Add necessary dependencies
        run: |
          helm repo add prefect https://prefecthq.github.io/prefect-helm
          helm repo update prefect

      - name: Create prefect namespace
        run: |
          cat <<EOF | kubectl apply -f -
          apiVersion: v1
          kind: Namespace
          metadata:
            name: ${{ inputs.namespace }}
          EOF

      - name: Replace default flow image
        run: |
          sed -i "s|DEFAULT_FLOW_IMAGE|${{ inputs.default_flow_image }}|g" ${{ inputs.helm_values_path }}

      - name: Run Helm upgrade commands
        run: |
          helm upgrade prefect-worker --install prefect/prefect-worker \
            -n ${{ inputs.namespace }} \
            --version ${{ inputs.prefect_chart_version }} \
            -f ${{ inputs.helm_values_path }}

Just like with creating a namespace, any missing resources can also be created. Usually, this includes a secret with the PREFECT_API_KEY and registry credentials secret for downloading private flow images, but additional secrets or configurations may also be needed.

After PR Marge

Tag and Release: A new GitHub tag and release are created for the updated version. Many ready-made actions are available online. For example, we can define a job like this:

jobs:
  prepare:
    if: ${{ github.event.pull_request.merged }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Get version from pyproject.toml
        id: get-version
        run: echo "version=$(grep -Po '(?<=^version = ")[^"]*' pyproject.toml)" | tee -a $GITHUB_OUTPUT

    outputs:
      VERSION: ${{ steps.get-version.outputs.version }}

  tag-and-release:
    if: ${{ github.event.pull_request.merged }}
    needs: [prepare]
    name: Tag and release new version (if applicable)
    runs-on: ubuntu-latest
    permissions:
      contents: write
    outputs:
      TAG_CREATED: ${{ steps.check-tag.outputs.exists != 'true' }}

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Check if a tag for this version already exists in the repo
        uses: mukunku/tag-exists-action@v1.6.0
        id: check-tag
        with:
          tag: v${{ needs.prepare.outputs.VERSION }}

      - uses: fregante/setup-git-user@v2
        if: steps.check-tag.outputs.exists != 'true'

      - name: Publish the new tag
        if: steps.check-tag.outputs.exists != 'true'
        run: |
          git tag -a v${{ needs.prepare.outputs.VERSION }} -m "Release v${{ needs.prepare.outputs.VERSION }}"
          git push origin v${{ needs.prepare.outputs.VERSION }}

      - name: Create a release
        if: steps.check-tag.outputs.exists != 'true'
        uses: ncipollo/release-action@v1
        with:
          generateReleaseNotes: true
          tag: v${{ needs.prepare.outputs.VERSION }}

We can utilize an additional prepare job that will pre-define values used later in our workflow, simplifying its logic.

Build PROD Image: Builds PROD image for the edp-flows, tagging it with ${VERSION} and pushing it to the GitHub container registry. The same template used for building the DEV image can be reused, but only the image tag can be changed.
Update DEV Prefect work pool: Updates the base job template in the DEV environment to use the new Docker image.
Update PROD Prefect work pool: Updates the base job template in the PROD environment to use the new Docker image.

Workflow 2: Prefect Worker Updates

This workflow is triggered by changes to any of the following files:

etc/helm_values/prefect-worker/values-dev.yaml
etc/helm_values/prefect-worker/values-prod.yaml

Pull Request Steps

Binary packages check/install: Installs any required binaries (like K3s and Helm) on the DEV virtual machine.
Update DEV Prefect work pool: Updates the Prefect worker’s base job template on DEV environment to apply any required configuration changes.

After PR Merge

Binary packages check/install: During the first execution, the necessary binaries (K3s and Helm) are installed on the PROD virtual machine.
Update DEV Prefect work pool: Updates the Prefect worker’s base job template on DEV environment to apply any required configuration changes.
Update PROD Prefect work pool: Updates the Prefect worker’s base job template on PROD environment to apply any required configuration changes.

Note: These first two workflows affect infrastructure only; they don’t touch actual Prefect deployments. The next workflow handles that.

Workflow 3: Prefect Deployment Orchestration

Triggered by changes to prefect.yaml or any source code within src directory.

Pull Request Steps

1. Identify Modified Deployments: In a simplified version, we can register all existing deployments. It can be handled with a helper script with such logic:

jobs:
  apply-deployments:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      
      - uses: actions/setup-python@v5
        with:
          python-version-file: ".python-version"

      - name: Install dependencies
        run: pip install -q PyYAML prefect

      - name: Get all deployments from prefect.yaml
        if: ${{ inputs.deployments == 'all' }}
        run: echo "DEPLOYMENT_NAMES=$(cat prefect.yaml | yq '.deployments[].name' | paste -sd ",")" | tee -a $GITHUB_ENV

As a target solution, we want a script that detects changes to deployments in the prefect.yaml by comparing the main branch with the pull request branch. This way, DEPLOYMENT_NAMES will include only the modified deployments. The script can also detect removed deployments, helping with housekeeping in Prefect Cloud.

2. Apply to DEV: Applies the modified deployments to the DEV Prefect workspace, referencing the pull request branch. Assuming that we have only modified deployments provided in DEPLOYMENT_NAMES, the registration script can look like this:

      - name: Set branch '${GITHUB_HEAD_REF}' in prefect.yaml
        run: |
          yq -i '.pull[] |= (select(has("prefect.deployments.steps.git_clone")) 
            | .["prefect.deployments.steps.git_clone"].branch = "${GITHUB_HEAD_REF}" | .) // .' prefect.yaml

      - name: Register all deployments
        run: |
              for deployment_name in $(echo ${{ env.DEPLOYMENT_NAMES }} | tr ',' '\n'); do

        prefect --no-prompt deploy \
            -n "$deployment_name"
            --tag ${GITHUB_HEAD_REF}
    done

Deployments are created with scheduling disabled to allow manual testing in the DEV environment. The preparation step sets GITHUB_HEAD_REF as a branch reference for the git_clone step at runtime.

3. Testing & Merge: Modified deployments and flows are tested. As there is no schedule enabled on the deployment on DEV workspace, it needs to be triggered and tested manually. Upon approval, changes are merged into the main branch.

After PR Merge

1. Identify Modified Deployments: Detects all updated deployments in the prefect.yaml, just like in step 1.

2. Apply to PROD: Apply the deployments to the PROD Prefect workspace, using the main branch. Scheduling is enabled with this command:

get_flow_name_for_deployment_from_prefect_yaml() {
    # Fetch flow name for a deployment from prefect.yaml.

    deployment_name=$1

    echo "Retrieving flow name for deployment '$deployment_name'..." >&2

    deployment_entrypoint=$(cat prefect.yaml | yq '.deployments[] | select(.name == "'$deployment_name'") | .entrypoint')
    flow_name=$(echo $deployment_entrypoint | cut -d':' -f2)
    flow_name_kebab_case=$(echo $flow_name | sed 's/_/-/g')

    # Return the flow name converted to kebab case, as this is what prefect CLI commands expect.
    echo $flow_name_kebab_case
}

flow_name=$(get_flow_name_for_deployment_from_prefect_yaml "$deployment_name")
schedule_id=$(prefect deployment schedule ls $flow_name/$deployment_name | grep '│' | awk -F '│' '{print $2}' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')
prefect deployment schedule resume $flow_name/$deployment_name $schedule_id

This script can be added to the workflow to automate schedule enabling.

3. Synv DEV: After deleting the pull request branch, reapply deployments to the DEV Prefect workspace, now referencing the main branch. Scheduling remains disabled for DEV.

You can verify the branch references used in deployments by checking the assigned tag or the Configuration tab in Prefect Cloud. For example, in the screenshot below, the deployment runs on branch feature_branch_1.

Conclusion & series summary

This article wraps up our four-part journey to building a modern, automated data platform—from high-level architecture to fully hands-off, production-ready operations. Along the way, we’ve shown how each layer, including architecture, infrastructure, orchestration, and automation, works together to create a resilient, scalable foundation for data engineering.

Let’s quickly recap:

Part 1 focused on architectural decisions, demonstrating how a lightweight Kubernetes setup on a single VM can enable rapid adoption and growth, even for teams just starting with cloud-native data platforms.

Part 2 moved from design to implementation, automating cloud infrastructure provisioning with Terraform to ensure consistency, reproducibility, and cloud-agnostic flexibility.

Part 3 took us deeper into operations, guiding you through containerized flow execution, Prefect worker configuration, and deployment management—helping you run your first data ingestion flows confidently.

And here in Part 4, we brought everything together by introducing CI/CD automation. We showed how three specialized workflows for Docker images, Prefect workers, and deployment orchestration help reduce manual errors, maintain a single source of truth, and scale both your platform and your team. This kind of automation makes development smoother, testing more reliable, and production releases faster and safer.

The main takeaway is that adopting CI/CD for data platforms is not just about tools but about changing how your team works. Automation connects development and production, reduces risk, and frees your engineers to focus on data rather than infrastructure.

Thanks so much for following along. For more tips and updates, check out my other articles, subscribe to our newsletter, and connect with me on LinkedIn. Let’s keep the conversation about smarter data platforms going.

Scaling Secure Data Access: A Systematic RBAC Approach Using Entra ID

2025-06-23T09:30:00Z

When data platforms grow in scale and complexity, so do the risks. Suddenly, you’re juggling dozens of tools, a growing number of users, and multiple layers of sensitive data, while trying to balance it all with security controls. Managing who gets access to what (and making sure they only get only what they need) quickly becomes a full-time job.

This reality requires flexible and manageable access controls. That’s exactly where Role-Based Access Control (RBAC) enters the scene.

What is Role-Based Access Control (RBAC)?

RBAC is a foundational security model that has evolved from military protocols to become a standard in IT systems. In this paradigm, user permissions are assigned to roles, rather than managing access for each user individually. This approach brings several advantages:

Scalability: Manage access for large user bases efficiently.
Consistency: Reduces the risk of misconfiguration.
Auditability: Makes it easier to track who has access to what.
Modularity: Aligns well with component-based data architectures.
Savings Cuts down on repetitive access management tasks.

At its core, RBAC secures both organizational assets and resources—a distinction that’s especially relevant in modular data platforms.

Assets are digital, valuable objects used across the organization to support data-driven decisions. These include data models, dashboards, reports, and machine learning models.

Resources are the underlying technical components or capabilities that enable the creation and delivery of those assets. These include EKS clusters, storage systems (e.g., S3, ADLS), databases, orchestrators, pipelines, and monitoring tools.

Access to both assets and resources should be managed consistently and securely. In most cases, Role-Based Access Control provides a solid and sufficient framework to do exactly that.

Why RBAC Matters in Modular Data Platforms

RBAC is essential in modular data platforms. These platforms serve diverse users (data engineers, analysts, scientists, AI engineers, and business users), each with specific access needs. The challenge lies in designing a system that grants appropriate access without compromising security or compliance.

This article walks through a structured, five-phase RBAC implementation using Microsoft Entra ID—a widely adopted identity and access management tool. The end goal of this process is to establish a streamlined, scalable access workflow, just like the one visualized below:

Phase 1: Defining User Personas

RBAC starts with clear role definitions, but many teams get stuck here. Where to start? Which roles do we need? What defines a good role? Let’s stick to 77/755/744…

This is a critical step, as your entire access model depends on getting this right.

The best approach begins by listing key user personas—broad categories of users with shared access needs, security considerations, and operational patterns—and refining them into sub-categories if necessary. Here are typical personas in a modern data platform:

Data Platform Engineer/DevOps – Needs broad access across infrastructure components. Responsible for developing, deploying, and maintaining platform services. Typically requires admin-level permissions across environments and tooling.
Data Engineer – Focuses on building and managing pipelines, data flows, and transformations. Requires read/write access in development and staging environments, limited access to production, and admin permissions in orchestration tools.
Data Scientist – Engages in exploratory data analysis and model development. Needs read access across multiple domains, access to computational resources, and controlled write access to model deployment tools.
Data Analyst – Primarily interacts with modeled or normalized data. Requires read access to curated datasets and limited write access for storing results. May need orchestrator access to refresh or trigger reporting workflows.
Business Analysts, BI Developers, and Report Creators – Work mostly in presentation layers. They typically require read-only access to business-ready data and tools for dashboard creation. Their access often spans multiple domains, which makes clear scoping important.

Pro tip: If your organization uses data domains (e.g., Finance, HR, Sales), use them as attributes to refine personas. A matrix of personas vs. domains can simplify access mapping without adding unnecessary complexity.

Phase 2: Creating EntraID Groups

With personas defined, the next step is to translate them into Microsoft EntraID security groups. They serve as the operational layer of the RBAC model, forming the backbone for scalable, repeatable permission management.

Entra ID groups do much more than just bundle users. They support critical capabilities for maintaining control as the platform scales, allowing audit trails, access reviews, and user lifecycle automation.

Each group should correspond to a specific persona, have a clearly defined owner, and include a domain context where applicable. For example, instead of creating a generic DataAnalyst group, consider a more specific DataAnalyst_Marketing group. Naming conventions should be meaningful, consistent, and designed to support future growth. A poorly named group today can create significant technical debt later if it needs to be split or redefined.

Security groups offer clear advantages over individual user permission assignments. They enable systematic and repeatable permission management that scales with organizational growth. When users move between roles or teams, administrators can adjust group memberships rather than managing permissions across individual services, reducing complexity, administrative overhead, and the risk of errors.

Some organizations also choose to create hierarchies of groups, where a parent group holds shared access, and child groups inherit those permissions while adding more specific scopes. This approach can work well if aligned with organizational structure, but it must be handled carefully to avoid unintentionally granting broader access than intended.

Phase 3: Mapping Users to EntraID Groups

With groups in place, the focus shifts to assigning users to appropriate groups based on their roles and domain access requirements, if applicable.

The principle of least privilege should guide all user assignments, ensuring that individuals receive only the minimum access necessary to perform their current job responsibilities. Start conservatively and expand access only when it’s validated by actual business need and approved by the data platform owner.

Many organizations struggle with group management and hesitate to adopt group-based assignments until they can fully automate the process. However, waiting for perfect automation can delay real security improvements. Even a manual group assignment following the audit rules is more secure and manageable than using user-resource access management.

Entra ID supports dynamic membership rules, which allow users to be automatically assigned to groups based on attributes like department, job title, or location. For example, analysts in the Finance department can be automatically added to the DataAnalyst_Finance group using:

(user.department -eq "Finance") and (user.role -contains "analyst")

These rules can also be extended by integrating Entra ID with external systems like Workday, allowing logic based on a richer organizational context to be applied.

Phase 4: Access Configuration and Permissions Setup

The access configuration phase translates group memberships into specific permissions across the modular data platform components. This phase requires a tools inventory, listing the components in use, and a data assets inventory, where the data catalog proves invaluable.

With the BoM and inventory in place, the next step is to enable EntraID as the identity provider. This can be achieved either through native support or via SSO, SAML, or OAuth. Once configured, RBAC is implemented at the component level by granting permissions to the appropriate groups.

This process should remain transparent to users while ensuring comprehensive logging for security monitoring and compliance purposes.

Phase 5: Audit and Continuous Monitoring

Regular auditing is essential for effective RBAC maintenance, ensuring that access permissions remain appropriate and aligned with organizational policies. This phase systematically reviews user assignments, group configurations, and access patterns to detect potential security risks or compliance issues. Organizations should define regular audit schedules, typically at monthly or quarterly intervals.

To maintain a robust RBAC setup, the following steps should be followed:

Step 1: Generate (new) User Overview

The Data Platform Owner initiates the generation of EntraID audit reports, which is typically carried out by the EntraID Team.
The Data Platform Owner verifies the BoM and access levels per role.
The resulting report is shared with the respective data and tool owners.

Step 2: Review User Overview

The Data Owner reviews the user access report.
They assess whether each user’s access is still appropriate.

Step 3: User Access Approval

If all user access is deemed appropriate (approval), the process concludes with a notification sent to relevant stakeholders.
If any user is rejected:
- The Data/Tool Owner requests that specific users be removed from relevant dashboards or reports.
- The affected users are notified of their access removal.

Step 4: Remove User Access

The EntraID Team removes the specified users from the corresponding EntraID Group.
A confirmation notification is sent to finalize the update.

Final Thoughts

Implementing RBAC in modular data platforms is not a one-time effort but a continuous, cyclical process. The five-phase methodology outlined here offers a scalable, repeatable framework that enhances security, reduces operational overhead, and improves transparency across the data ecosystem.

With Microsoft Entra ID as the backbone, organizations gain enterprise-grade identity management, automation support, and auditable group control. It enables sustainable, compliant access control by supporting the full user and permission lifecycle. When paired with regular audits, this approach ensures that RBAC remains aligned with evolving business needs and security risks, with audit findings feeding back into earlier phases for ongoing refinement.

Getting to Your First Flow Run: Prefect Worker & Deployment Setup

2025-06-10T11:00:00Z

You’ve laid the groundwork: the infrastructure is in place. The next logical step is turning that foundation into something functional, running your first data ingestion workflow. That moment when everything connects for the first time can feel like crossing an invisible line: from setup to real-world execution.
This article picks up where we left off. It’s the third part in a series designed to guide data engineers through the complete journey of building a modern data platform.

In Part 1, I explored architectural approaches and proposed a lightweight Kubernetes setup running on a single VM. While it doesn’t offer full high availability, this setup has proven to be a practical starting point, especially for teams with limited cloud-native experience. It allows organizations to grow along the data maturity curve without the overhead of more complex solutions.

Part 2 focused on provisioning the infrastructure using Terraform on Google Cloud Platform (GCP). We used GCP as an example, but the underlying architectural principles are cloud-agnostic and applicable across providers.

Now that the infrastructure is ready, this article walks through the next milestone: configuring all the components required to execute your first data ingestion workflow.

Data Platform Components Overview

Here’s a high-level overview of a generic data platform architecture (as discussed in previous articles).

While this article won’t cover the data warehouse or data lake, we will zoom in on the other key components that power workflow orchestration:

GitHub as the version control system
Prefect Cloud as the orchestration environment
Prefect worker as the workflow orchestration system

To prepare these, we’ll walk through these essential elements that serve as the backbone of a modern data platform:

Docker Image for Flows Execution

Every Prefect flow needs a controlled and consistent environment to run. Using a Docker container is an ideal solution for this purpose, as it provides isolation and ensures that all dependencies and runtime configurations are reproducible. Containerization is a standard practice in modern data platforms to guarantee reliable flow execution across different environments.

Prefect Worker

Prefect Cloud handles all deployment schedules, but a process within our infrastructure is required to pull and execute these scheduled tasks. This is the responsibility of the Prefect worker. Before diving into more advanced topics, it’s important to understand how to configure and manage the Prefect worker. Later, I will explain further how to set up a base job template that will be used to execute all deployments, ensuring consistent and scalable workflow orchestration.

Prefect Configuration Files

A well-structured repository is essential for managing deployments and flows efficiently. Ideally, adding a new deployment should require only a few lines of code, making it easy for any team member to contribute. The prefect.yaml file plays a key role in this process by organizing and codifying deployment configurations, clearly connecting flows, deployments, and infrastructure in a maintainable way.

By focusing on containerized execution, robust workflow orchestration, and clear configuration management, we lay the groundwork for a scalable and maintainable data platform that ensures reliable data ingestion and processing.

Docker Image for Flows Execution

Having each Prefect flow executed in its own isolated environment is essential. Without isolation, two flows running side-by-side could conflict—think mismatched library versions, breaking changes, or dependency clashes. Suddenly, what worked yesterday doesn’t work today. Containerization solves this problem elegantly.

Building a dedicated Docker image ensures that flow runs are reproducible and decoupled from the host system. You can test new dependencies in dev, tag the image, and confidently promote it to prod. No more “it worked on my machine” surprises.

For most data teams, starting with the official Prefect image is the way to go. It includes all the Prefect orchestration tools out of the box, so you don’t have to reinvent the wheel. Manage dependencies via uv and a pyproject.toml:

dependencies = [
    "prefect[docker,github,gcp]>=3.3.7",
    "dlt[mssql,parquet]>=1.8.1",
    "pymssql>=2.3.2",
]

Once dependencies are set, run uv sync. This generates a uv.lock file, locking all versions for reproducibility. This file, along with your pyproject.toml, is all you need for the build process.

The Dockerfile itself is rather straightforward, assuming we need to prepare the uv to install system dependencies to be used inside the container without an additional virtual environment:

FROM prefecthq/prefect:3.3.7-python3.12

RUN pip install uv --no-cache

COPY pyproject.toml uv.lock README.md ./

RUN uv export --frozen --no-dev --no-editable > requirements.txt

RUN uv pip install --no-cache --system --pre -r requirements.txt

Notice we don’t copy the entire repo, only dependency files. The flow code lives outside the image, so we only rebuild the image when dependencies change. To push the image to the GitHub Container Registry, you can use:

docker build -t ghcr.io/<your_github_organisation>/edp-flows:<tag> .

docker push ghcr.io/<your_github_organisation>/edp-flows:<tag>

This is a manual step for now, but don’t worry—we’ll automate it with GitHub Actions soon. For now, you’ve got a solid, reliable foundation for running flows in a clean, isolated environment every single time.

Prefect Worker

Once the Docker image is ready, the next step is orchestrating flows execution. That’s where the Prefect worker comes in. Workers are long-running processes that poll work pools for scheduled flow runs and execute them. If you want to dive deeper into the range of possibilities, Prefect’s documentation is a great resource.

For our setup, we’ll use the Kubernetes worker type and deploy it with the official Prefect Helm Chart.

How Prefect Executes Flow Runs

Prefect workers are responsible for:

Polling work pools for scheduled flow runs
Spinning up job-specific infrastructure using base templates
Enabling environment-specific configurations via Helm charts
Supporting zero-downtime updates when modifying worker settings

This dynamic approach means you can flexibly scale, update, and manage your workflow execution environments.

Prefect Worker Configuration

Here’s a minimal values.yaml example, sufficient to get a Prefect worker running via Helm:

worker:
  image:
    repository: prefecthq/prefect
    prefectTag: 3.3.7-python3.12-kubernetes
    pullPolicy: IfNotPresent

  config:
    workPool: "edp-work-pool"
    workQueues: ["default"]
    name: "edp-worker"
    baseJobTemplate:
      configuration: << BASE JOB TEMPLATE WILL BE PROVIDED IN THE NEXT SECTION >>

  cloudApiConfig:
    accountId: "7e3c367b-143a-86e2-b92f-6i414816c39b"
    workspaceId: "c506815f-qe83-42dc-b905-re6bcfb68c52"
    apiKeySecret:
      name: prefect-api-key-secret
      key: PREFECT_API_KEY

Each of the three key sections plays a vital role:

Image: Specifies the Prefect image used to run the worker. This is not the flow’s image that runs your business logic; that will be defined in the baseJobTemplate.
Config: Defines work pool, queues, worker name, and most importantly, the baseJobTemplate. An example of a working base job template is shown in the next section.
cloudApiConfig: Provides the cloud workspace details, such as account and workspace IDs. You’ll also need to configure a service account in Prefect Cloud and store its API token as a Kubernetes secret (prefect-api-key-secret with the key PREFECT_API_KEY).

Example Base Job Template

In a Kubernetes environment, the base job template defines how Prefect spins up infrastructure for each flow run. The minimal job template for a single-pod job below will allow you to set:

The job name (helpful for distinguishing jobs and pods in Kubernetes)
The image tag used in the pod
The namespace for job creation
Other Kubernetes settings include image pull secrets, retry limits, and job cleanup.

{
  "variables": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string",
        "title": "Name given to job created by a worker (key: name)",
        "default": "edp-k8s-job"
      },
      "image": {
        "type": "string",
        "title": "Image name and tag that will execute flows, to be provided from deployment (key: image)",
        "default": "ghcr.io/<your_github_organisation>/edp-flows:<tag>"
      },
      "namespace": {
        "type": "string",
        "title": "Namespace name where jobs will be scheduled (key: namespace)",
        "default": "prefect"
      }
    }
  },
  "job_configuration": {
    "env": {},
    "name": "{{ name }}",
    "labels": {},
    "command": "",
    "namespace": "{{ namespace }}",
    "job_manifest": {
      "kind": "Job",
      "spec": {
        "template": {
          "spec": {
            "volumes": [],
            "containers": [
              {
                "env": [],
                "name": "prefect-job",
                "image": "{{ image }}",
                "imagePullPolicy": "Always",
                "envFrom": [],
                "volumeMounts": []
              }
            ],
            "completions": 1,
            "parallelism": 1,
            "tolerations": [],
            "restartPolicy": "Never",
            "imagePullSecrets": [
              {
                "name": "reg-creds"
              }
            ]
          }
        },
        "backoffLimit": 0,
        "ttlSecondsAfterFinished": 7200
      },
      "metadata": {
        "namespace": "{{ namespace }}"
      },
      "apiVersion": "batch/v1"
    },
    "stream_output": true
  }
}

You can customize this further as needed, and include the template in your Prefect worker configuration. To deploy a worker, it’s enough to run the helm command:

helm upgrade prefect-worker --install prefect/prefect-worker \
            -n prefect \
            -f ${{ helm_values_path }}

Once deployed, your worker should appear in the Work Pool section in Prefect Cloud.

The baseJobTemplate is also exposed as a config map in your Kubernetes cluster. To follow best practices, manage all worker configuration changes as infrastructure-as-code. Use GitHub workflows to apply updates automatically, reducing the risk of human error from manual changes in Prefect Cloud.

For more security, assign the service account used to register the worker the “Developer” role, and limit regular developers to read-only access within the Work Pool. These permission settings can be configured in your Prefect Cloud account settings:

Once your Prefect worker is up and running, you’re ready to register your first deployment.

Prefect Configuration Files

The prefect.yaml file describes base settings for all deployments, with additional instructions for preparing the execution environment for a deployment run. It can be initialized with the prefect init command, and after filling in the data, you might end up with a file like this:

name: prefect-deployments
prefect-version: 3.3.7

definitions:
  schedules:
    cron_default: &cron_default
      cron: "0 0 * * *"
      timezone: "UTC"
      active: false

build: null
push: null

pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect
  - prefect.deployments.steps.git_clone:
      repository: https://github.com/<your_github_organisation>/edp-flows.git
      access_token: "{{ prefect.blocks.github-credentials.edp-github-credentials.token }}"
  - prefect.deployments.steps.run_shell_script:
      directory: "/opt/prefect/edp-flows"
      script: |
        uv pip install --no-cache --system .

deployments:
  - name: hello_world
    description: "Test hello-world deployment."
    schedules:
      - <<: *cron_default
        cron: "5 0 * * *"
    entrypoint: src/edp_flows/flows/hello_world.py:hello_world_flow
    parameters:
      text: "Hello, world!"

The most notable sections are:

Definitions: Shared properties such as schedules.
build/push: Relevant only if you need to build a fresh image with each deployment; not applicable in our case (Docker images are maintained separately from prefect.yaml.)
Pull: Clones the repository to ensure each deployment uses the latest code. During development, it can be configured to target specific branches. Finally, it installs all Python modules so they’re available during execution.
Deployments: A structured list of deployments, each with customizable parameters, allowing the same flow to be reused across multiple scenarios.

Example Prefect Flow

As Prefect flows aren’t covered in detail here, we’ll use a simple “Hello, World!” example for illustration. In your actual use case, this is where you would implement the logic for your first ingestion workflow, tailored to your specific data source and target (such as a data warehouse or lake). Here is hello_world.py:

"""A flow to demonstrate how to log messages in Prefect flows."""

from prefect import flow, get_run_logger, task

@task(log_prints=True)
def print_log_prints(text: str) -> None:
    """Attempt to print text, world using regular Python print() function.

    This time, use the `log_prints` task parameter.
    """
    print(f"log_prints=True: {text}")  # noqa: T201

@task
def log_prefect_run_logger(text: str) -> None:
    """Attempt to log text, world using Prefect's runtime logger."""
    logger = get_run_logger()
    logger.info(f"Prefect runtime logger: {text}")


@flow
def hello_world_flow(text: str) -> None:
    """Demonstrate how to log messages in Prefect flows."""
    print_log_prints(text)
    log_prefect_run_logger(text)

prefect --no-prompt deploy \
        --name "$deployment_name" \
        --tag $tag \
        --pool $work_pool \
        --job-variable name=$deployment_name

The registered deployment includes a schedule, but it’s disabled by default. To enable it, either do so manually in Prefect Cloud or use the following commands:

prefect deployment schedule ls <flow_name>/<deployment_name>
prefect deployment schedule resume <flow_name>/<deployment_name>

After running the first command, you should see a view like this:

After enabling the schedule, it should appear as Active, and the Prefect worker will trigger it every day at noon. To test a new deployment, you can manually trigger it from Prefect Cloud. Once your deployment runs successfully, you will see logs from it like the following:

Conclusion

With this setup, you now have everything in place to execute your first data ingestion flow. You’ve:

Built a containerized flow execution environment
Deployed a scalable Prefect worker
Defined clean, reusable deployment configurations.

This architecture gives you flexibility and control across environments and sets the stage for more advanced workflows.

While the example used here is a simple “Hello World!” flow, the same deployment structure can be applied to your real data ingestion workflows. To run your first actual ingestion pipeline, all you need to do is replace the flow logic with code that connects to your data source and writes to your destination (like a data warehouse or lake). The orchestration, environment, and deployment pieces remain the same.

What’s next?

In the next article, I’ll show you how to automate this entire process using GitHub Actions, turning this manual setup into a streamlined CI/CD pipeline your whole team can rely on.

Roles in the Context of the Analytics Workflow

2025-03-27T10:47:00Z

An analytics workflow documents the journey from raw data to production-ready data models, encompassing development and testing phases. A critical component is governance, implemented through a Pull Request approval process that facilitates regular code reviews and prevents technical debt accumulation. This structured approach ensures quality and maintainability while supporting collaborative development.

To manage this governance effectively, several roles are typically involved. It’s becoming increasingly rare to see a single analyst handle the entire end-to-end process of creating a report or data model. Instead, the trend is moving toward a growing number of specialized roles, each with distinct responsibilities.

Key Roles

The analytics workflow involves several key roles, with data analysts playing a particularly key position. Data analysts combine technical skills with deep business domain knowledge, giving them unique insight into business models and challenges.

In contrast, data engineers and data platform engineers typically focus on technical implementation rather than direct business interaction. While aligning the data platform roadmap and investments with business value remains strategically important for leadership, this alignment happens at a higher level and doesn’t directly impact the day-to-day analytics workflow operations.

Role	Tasks
Data Analyst	- Understand business requirements - Analyze data in intermediate and mart layers - Develop SQL queries and transformations - Create and maintain metadata documentation
Data Platform Engineer	- Monitor and support infrastructure resources - Maintain CI/CD pipelines - Manage network infrastructure - Implement cybersecurity measures
Data Engineer	- Design and develop data pipelines - Maintain and optimize data flows - Schedule and orchestrate data processing - Implement data ingestion processes

Measuring the Impact of Each Role

Because roles usually tackle different problems, it is a good idea to measure performance and impact differently. Measuring how a role is doing is also important for creating rules such as notification rules, issue and incident prioritization rules, and other operational matters to increase the reliability of production.

Role	Measure
Data Analyst	- Understanding of business domain - Business Satisfaction - ROI from data initiatives
Data Platform Engineer	- Speed of new data platform features - Reliability of the data platform - Data analysts’ support and satisfaction
Data Engineer	- Speed of new data ingestions - Reliability of data pipelines - Data analysts’ support and satisfaction

Understanding the Need for Roles

Analytics teams often tend to combine roles or leave them loosely defined. This approach is understandable, and sometimes even beneficial, in the early stages of an analytics initiative. After all, when starting out, the priority is delivering business value quickly, and formal roles and approval processes can slow things down.

However, this lack of clearly defined roles and boundaries typically creates challenges as the analytics function matures. Common issues include:

Blurred lines between exploratory analytics work and production pipeline operations make it difficult to maintain service levels
Insufficient knowledge transfer mechanisms, including limited documentation, unclear onboarding processes, and a lack of backup coverage for key roles
Team friction arising from ambiguous responsibilities and overlapping ownership
An overemphasis on technical tools and implementation details, rather than addressing the more fundamental needs of role clarity and process alignment

Working With an External Partner

In this workflow, there are already quite a few roles and tasks, and in reality, even more can be involved. For example, handling data privacy in Europe requires a solid understanding of GDPR. Given the range of responsibilities and the breadth of expertise needed to run a data department effectively, many data and analytics teams choose to rely on external partners. However, this isn’t always straightforward. External teams often create strong and rigid boundaries between themselves (the extended team) and the customer’s in-house team (the internal team), which can hinder collaboration.

One practical way to navigate the tension between collaboration and rigid boundaries is to establish a clear RACI matrix from the very beginning. This matrix serves as a shared reference to define roles and responsibilities, helping both internal and extended teams understand who is Responsible, Accountable, Consulted, and Informed for each task. It provides structure without creating silos, enabling smoother handovers and aligned expectations.

Conclusions

A well-structured analytics workflow is essential for turning raw data into reliable insights. As analytics initiatives mature, the need for governance, role clarity, and collaboration becomes increasingly important. Specialized roles such as data analysts, data engineers, and platform engineers each contribute unique expertise, and their impact should be measured differently to reflect their responsibilities.

Clearly defined roles and processes—such as code reviews, CI/CD practices, and RACI matrices—not only support governance and maintainability but also foster collaboration across internal and external teams. While early-stage flexibility is useful, long-term success in data and analytics depends on thoughtful structure, cross-functional alignment, and a shared understanding of who does what and why.

Resources

Example of RACI Matrix

How to Setup Data Platform Infrastructure on Google Cloud Platform with Terraform

2025-03-05T13:31:00Z

Setting up a solid, scalable data platform is crucial for organizations looking to get the most out of their data. Building upon our previous discussion on architectural considerations for deploying Prefect on various cloud platforms, this article will walk you through building your data platform infrastructure on Google Cloud Platform (GCP) using Terraform.

Our focus is on creating a server-based approach utilizing a single Virtual Machine (VM)—a simple yet powerful starting point for organizations that don’t need to dive into complex source systems or full-blown data warehouses just yet. This approach offers an easy entry point with plenty of room to grow as your data needs evolve.

As this article will use a substantial amount of code, you can find all the relevant code samples in our dedicated repository.

Why Choose a Server-Based Approach with a Single VM?

Choosing a server-based approach with a single VM comes with several advantages:

Cost-effectiveness: A single VM setup is often a more budget-friendly option for initial deployments or smaller-scale projects.
Simplified management: Fewer components mean easier maintenance and troubleshooting.
Flexibility: This approach offers the ability to easily expand or modify your infrastructure as requirements change.
Learning curve: For teams new to cloud infrastructure, starting with a single VM can be less overwhelming and serve as a stepping stone to more complex architectures.

This guide will walk you through the process of setting up key components of our data platform infrastructure on GCP. You’ll learn how to configure the VPC and subnets, set up Compute Engine instances, configure firewall rules, secure SSH access with Identity-Aware Proxy (IAP), establish internet connectivity with Cloud Router and NAT, and store the state files in Cloud Storage. We’ll also dive into the specifics of GCP’s Identity-Aware Proxy, exploring its crucial role in enhancing the security of our data platform.

By using Terraform to manage infrastructure as code, we ensure that our setup is reproducible, version-controlled, and easy to manage. This not only streamlines the initial deployment but also makes scaling and future updates much more efficient.

Let’s get started on building a solid, scalable data platform infrastructure on GCP—one that will grow with your organization’s data needs.

Infrastructure Overview

Before we dive into the step-by-step process of setting up your data platform on GCP, let’s take a first look at the key components that make up the environment we’ll be building:

Virtual Private Cloud (VPC): A private network that will serve as the foundation of your environment, providing isolation and security.
Subnet: A private subnet where the virtual machine will reside.
Compute Engine Virtual Machine: The instance where both the GitHub Runner and Prefect Worker will be set up.
Firewall: Configured with rules to allow inbound access exclusively through Google Cloud Identity-Aware Proxy (IAP), blocking all other traffic.
IAP SSH Permissions: Enables secure access to the virtual machine.
Cloud Router: Provides internet connectivity for the virtual machine.
Cloud NAT: Configures a NAT gateway that directs the virtual machine to the Cloud Router for outbound internet access. It also ensures that the public IP is fixed as long as the Cloud NAT object is not destroyed and configured for the same zone.
Cloud Storage: Sets up a Google Cloud Storage bucket to store ingested data as Parquet files before transforming and loading it into the database as tables.

Understanding Google Cloud Identity-Aware Proxy (IAP)

To ensure a secure environment, all public access should be completely blocked. With this configuration, resources within the environment can be accessed using two main options:

VPN Connection: In this setup, at least one resource within the VPC must be exposed to the internet to host a VPN endpoint. Alternatively, a separate VPC can be configured solely for VPN purposes, with VPC Network Peering into the main environment. This way, only the VPN-hosting VPC is exposed to the internet, while the main environment remains accessible only internally. Although effective, this configuration is more complex and falls outside the scope of this documentation.
Google Cloud Identity-Aware Proxy (IAP): This option offers a similar secure access model to a VPN but with simplified management through Google Cloud. As outlined in the official documentation:

“When an application or resource is protected by IAP, it can only be accessed through the proxy by principals, also known as users, who have the correct Identity and Access Management (IAM) role. When you grant a user access to an application or resource by IAP, they’re subject to the fine-grained access controls implemented by the product in use without requiring a VPN. When a user tries to access an IAP-secured resource, IAP performs authentication and authorization checks.”

This diagram from Google further illustrates the components required to implement this configuration:

With a solid understanding of Google Cloud Identity and its role in managing users and access, let’s now dive into the practical steps for setting it up and implementing it effectively.

Phase 1: Securing the Essentials

Before starting the Terraform configuration, make sure you have the following tools and setups in place:

Terraform: You’ll need Terraform installed on your local machine. This will be your main tool for provisioning infrastructure.
gcloud CLI: The gcloud CLI tool should be installed and configured to interact with your Google Cloud account.
GCP Service Account: A Google Cloud Platform service account needs to be created.
Enabled APIs: Make sure the Compute Engine API and Cloud Resource Manager API are enabled on your GCP account.

Let’s take a detailed look at these prerequisites:

Step 1: Installing Terraform

Terraform can be installed in various ways, which are outlined by Hashicorp here. For Ubuntu, installation can be done with the following commands:

wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o
/usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg]
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee
/etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

Step 2: gcloud CLI installation

Similarly to Terraform, the gcloud CLI can be installed as per the official instructions. For Ubuntu, run:

 
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates gnupg curl
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo gpg --dearmor
-o /usr/share/keyrings/cloud.google.gpg
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg]
https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a
/etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update && sudo apt-get install google-cloud-cli

After installation, initialize the gcloud CLI by providing the “gcloud init” command and setting up a new account by opening the provided URL.

gcloud init


# gcloud init
Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
  gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

You must sign in to continue. Would you like to sign in (Y/n)?  Y

Go to the following link in your browser, and complete the sign-in prompts:

    https://accounts.google.com/o/oauth2/auth<URL_TO_OPEN_IN_BROWSER>

Once finished, enter the verification code provided in your browser:
<PROVIDE_VERIFICATION_CODE>

Subsequently, configure the desired Cloud project, default Compute Region, and Zone for your environment.

Step 3: Setting up GCP Service Account

To create a GCP Service Account, navigate to the Google Cloud Console, select the correct project, and go to Navigation Menu (3 lines) > IAM & Admin > Service Accounts.

Click Create Service Account and provide the required information.

When prompted to Grant this service account access to a project, select the appropriate role. In this guide, we use the Owner role for simplicity, but it’s advisable to limit permissions to only what’s necessary.

Finally, in the Grant users access to this service account step, assign access to the users who will need to interact with the Kubernetes cluster (not part of this article) or VM. Once done, verify that the service account is correctly set up. Its email should follow this pattern:

{service_account_name}@{project}.iam.gserviceaccount.com

Step 4: Downloading the JSON Key for the Service Account

In the service account interface, go to Actions (3 dots) > Manage keys.

Select Add Key > Create new key > JSON to download the JSON key file. Keep this file secure, as it will be required for Terraform configuration.

Step 5: Activating the Service Account

After downloading the JSON key, activate the service account locally with the following command:

gcloud auth activate-service-account
{service_account_name}@{project}.iam.gserviceaccount.com
--key-file={json_file}.json

This step enables the service account for use in the local environment, ensuring access to necessary GCP resources with IAP tunnel functionality.

Step 6: Generating HMAC Key to Buckets

To enable file uploads to Cloud Storage, some libraries require an HMAC token in addition to a JSON key. To generate an HMAC token:

Go to Cloud Storage > Settings > Interoperability.
Under Service account HMAC, click Create a key for service account.
Once created, the token will be marked as ‘Active’.

For basic configuration, this step is not required. However, for ingestion, the key must be added to Google Secret Manager to ensure it’s accessible for flow runs.

Step 7: Enabling Compute Engine API and Cloud Resource Manager API

Before Terraform can interact with GCP, make sure these APIs are enabled for your project:

If they’re not enabled, head to the Google Cloud Console and enable them.

Step 8: Setting Up a Remote State for Terraform

By default, Terraform stores its state locally in .tfstate files. While this works for development, for any persistent environment, even if you’re the only one working on the project, it’s crucial to store this state in a centralized and reliable location. A common best practice is to use a Google Cloud Storage (GCS) bucket to keep the state safe and accessible, avoiding potential issues with local file loss or conflicts.

However, Terraform itself cannot create the bucket required for storing its state, leading to what’s called a “chicken-and-egg” problem. The bucket must be created manually before running any Terraform code. Tools like Terragrunt can solve this by simplifying environment management and reducing code duplication (example setup). However, for the sake of simplicity, we are not introducing such tools in this context.

To create a new bucket using the gcloud CLI, export the necessary credentials and then proceed with the bucket creation process.

Step 9: Exporting Credentials and Setting up a New Bucket

Once we have our service account JSON prepared, an export of credentials is necessary to provide Application Default Credentials (ADC):

export GOOGLE_APPLICATION_CREDENTIALS=test-project-32206692d146.json

Then, with the usage of gcloud CLI, a new bucket with the applied policy should be created:

gcloud storage buckets create gs://test-project-tfstate --location=us-central1 
--uniform-bucket-level-access

gcloud storage buckets add-iam-policy-binding gs://test-project-tfstate \
--member="serviceAccount:test-service-account@test-project.iam.gserviceaccount.com" \
  --role="roles/storage.objectAdmin"

Once completed, it should be available in the GCP Console. To check it, go to Navigation Menu > Cloud Storage > Buckets:

Phase 2: Installing & Deploying Infrastructure with Terraform

To set up the environment, Terraform will handle provisioning all the required GCP resources. By the end of this process, your directory structure will look like this:

$ tree

|-- backend.tf
|-- main.tf
|-- provider.tf
|-- test-project-32206692d146.json
|-- variable.tf

For managing both DEV and PROD environments, you can duplicate the files as shown:

$ tree
├── dev
    ├── backend.tf
│   ├── main.tf
│   ├── test-project-32206692d146.json
│   ├── provider.tf
│   └── variable.tf
└── prod
    ├── backend.tf
    ├── main.tf
    ├── test-project-32206692d146.json
    ├── provider.tf
    └── variable.tf

Terraform Files

The main difference between environments lies in the backend.tf and variables.tf files. In larger projects, using Terraform modules or tools like Terragrunt is recommended for reusable configurations. However, for simplicity, this example uses code duplication, which is also a valid approach.

The content of provider.tf should look like this:

provider "google" {
  region      = var.region
  project     = var.project_name
  credentials = file(var.credentials_file)
  zone        = var.zone
}

backend.tf should point to a bucket with a shared tfstate file created in step 9 of the first phase. It needs to be manually configured because it is the first block loaded when running terraform init, and variables from variables.tf cannot be referenced here:

terraform {
  backend "gcs" {
    bucket  = "test-project-tfstate"
    prefix  = "terraform/state/prod"
  }
}

All variables used in provider.tf and main.tf are defined in variable.tf, as shown below:

variable "credentials_file" {
  default = "test-project-32206692d146.json"
}

variable "environment" {
  default = "prod"
}

variable "filesystem" {
  default = "ext4"
}

variable "image" {
  default = 
"projects/ubuntu-os-cloud/global/images/ubuntu-2404-noble-amd64-v20241115"
}

variable "ip_cidr_range" {
  default = "10.202.0.0/24"
}

variable "machine_type" {
  default = "c3d-standard-8-lssd"
}

variable "project_name" {
  default = "test-project"
}

variable "region" {
  default = "us-central1"
}

variable "service_account" {
  default = 
"serviceAccount:test-service-account@test-project.iam.gserviceaccount.com"
}

variable "zone" {
  default = "us-central1-c"
}

main.tf defines and initializes all infrastructure components outlined in the Infrastructure overview section.

resource "google_compute_network" "vpc_edp" {
 name                    = "vpc-${var.project_name}-${var.environment}"
 auto_create_subnetworks = "false"

}

resource "google_compute_subnetwork" "subnet_edp" {
 name          = "subnet-${var.project_name}-${var.environment}"
 ip_cidr_range = var.ip_cidr_range
 network       = google_compute_network.vpc_edp.name
 region        = var.region
 depends_on    = [google_compute_network.vpc_edp]
}

resource "google_compute_instance" "vm_edp" {
 project      = var.project_name
 zone         = var.zone
 name         = "${var.project_name}-${var.environment}-01"
 machine_type = var.machine_type
 boot_disk {
   auto_delete = true
   initialize_params {
     image = var.image
     size  = 50
     type  = "pd-ssd"
   }
   mode = "READ_WRITE"
 }
 scratch_disk {
   interface = "NVME"
 }
 network_interface {
   network    = "vpc-${var.project_name}-${var.environment}"
   subnetwork = google_compute_subnetwork.subnet_edp.name
 }
 metadata_startup_script = <<-EOT
   #!/bin/bash
   set -e
   sudo mkfs.ext4 -F /dev/disk/by-id/google-local-nvme-ssd-0
   sudo mkdir -p /mnt/disks/local-nvme-ssd
   sudo mount /dev/disk/by-id/google-local-nvme-ssd-0 /mnt/disks/local-nvme-ssd
   sudo chmod a+w /mnt/disks/local-nvme-ssd

   echo UUID=`sudo blkid -s UUID -o value /dev/disk/by-id/google-local-nvme-ssd-0` /mnt/disks/local-nvme-ssd ext4 discard,defaults,nofail 0 2 | sudo tee -a /etc/fstab
 EOT
 depends_on              = [google_compute_network.vpc_edp]
}

resource "google_compute_firewall" "rules" {
 project = var.project_name
 name    = "allow-ssh-${var.environment}"
 network = "vpc-${var.project_name}-${var.environment}"

 allow {
   protocol = "tcp"
   ports    = ["22", "6443"]
 }
 source_ranges = ["35.235.240.0/20"]
 depends_on    = [google_compute_network.vpc_edp]
}

resource "google_project_iam_member" "project" {
 project = var.project_name
 role    = "roles/iap.tunnelResourceAccessor"
 member  = var.service_account
}

resource "google_compute_router" "router" {
 project    = var.project_name
 name       = "nat-router-${var.environment}"
 network    = "vpc-${var.project_name}-${var.environment}"
 region     = var.region
 depends_on = [google_compute_network.vpc_edp]
}

resource "google_compute_router_nat" "nat" {
 name                               = "router-nat-${var.project_name}-${var.environment}"
 router                             = google_compute_router.router.name
 region                             = var.region
 nat_ip_allocate_option             = "AUTO_ONLY"
 source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

 log_config {
   enable = true
   filter = "ERRORS_ONLY"
 }
}

resource "google_storage_bucket" "private_bucket" {
 name          = "${var.project_name}-${var.environment}"
 location      = var.region
 storage_class = "STANDARD"

 uniform_bucket_level_access = true
}

resource "google_storage_bucket_iam_binding" "bucket_writer" {
 bucket = google_storage_bucket.private_bucket.name

 role = "roles/storage.objectCreator"
 members = [
   "${var.service_account}"
 ]
}

resource "google_storage_bucket_iam_binding" "bucket_admin" {
 bucket = google_storage_bucket.private_bucket.name

 role = "roles/storage.admin"
 members = [
   "${var.service_account}"
 ]
}

With the environment outlined, let’s provision the infrastructure by validating, formatting, and applying the configuration.

Step 1: Infrastructure Provisioning with Terraform

Once the necessary files are prepared, validate and format the configuration:

$ terraform validate
Success! The configuration is valid.
$ terraform fmt
main.tf
provider.tf

Before applying changes, inspect them with the plan command:

terraform plan
# Check if it's all good
terraform apply
# Enter a value: yes

After a few minutes, the environment will be ready. To list the created resources, run:

$ terraform state list
google_compute_firewall.rules
google_compute_instance.vm_edp
google_compute_network.vpc_edp
google_compute_router.router
google_compute_router_nat.nat
google_compute_subnetwork.subnet_edp
google_project_iam_member.project

Step 2: Set up Verification

Verify the resources by logging into the GCP Console. Confirm the creation of VPC and Subnet, Virtual Machine, Firewall Rule, IAP SSH Permission, Cloud Router, and NAT Gateway. Navigate to the following sections:

VPC and Subnet

Go to Navigation Menu > VPC Network > VPC Networks:

Click on the VPC and check the Subnets tab:

Virtual Machine

Navigate to Navigation Menu > Compute Engine > VM instances:

Firewall Rule

Go to Navigation Menu > VPC Network > Firewall:

IAM Role

Navigate to Navigation Menu > IAM & Admin > IAM, and View by roles:

Cloud NAT gateway

Go to Navigation Menu > Network Connectivity > Cloud Routers > Open Cloud Router:

Cloud NAT

Navigate to Navigation Menu > Network Services > Cloud NAT:

SSH Access to Virtual Machine using GCP Console

To test SSH connectivity to the Virtual Machine, go to Navigation Menu > Compute Engine > VM instances and click on the SSH option for the created VM. Approve the connection when prompted, and you should be logged in.

This verification ensures all components are correctly configured and accessible. The next steps in setting up the data platform should be setting up a self-hosted GitHub runner and then a Prefect worker.

SSH Access to Virtual Machine Using the gcloud CLI

You can access a Virtual Vachine securely using only a service account token and gcloud CLI. Follow these steps to set up and establish SSH access:

Ensure you have the service account JSON key stored locally.
Log in to your Google Cloud account using the following command:

gcloud auth activate-service-account 
test-service-account@test-project.iam.gserviceaccount.com --key-file
 ~/.config/gcloud.json
gcloud config set project test-project

Once authenticated, execute the following command to initiate the SSH connection:

gcloud compute ssh ${project_name}-${environment}-01

On the first execution, the gcloud CLI will prompt you to generate a private and public SSH key pair. Follow the instructions to create the key pair.

Once the keys are created, access to the virtual machine will be automatically established. Subsequent logins will reuse the existing key pair, simplifying future access.

Conclusion

Setting up a data platform infrastructure on Google Cloud Platform using Terraform provides a solid foundation for organizations looking to leverage the power of their data. This approach offers several key benefits:

Scalability and Flexibility: The server-based approach with a single VM provides an excellent starting point that can easily be expanded as your data needs grow.
Security: By leveraging Google Cloud’s Identity-Aware Proxy (IAP), we’ve ensured that access to our resources is tightly controlled and secure.
Infrastructure as Code: Using Terraform allows for version-controlled, reproducible infrastructure deployments, making it easier to manage and update your environment over time.
Cost-Effectiveness: Starting with a single VM setup is often more budget-friendly for initial deployments or smaller-scale projects.
Simplified Management: With fewer components to manage initially, maintenance and troubleshooting become more straightforward.

By following the steps outlined in this guide, you’ve created a robust infrastructure that includes a VPC, subnet, Compute Engine instance, firewall rules, IAP SSH permissions, Cloud Router, Cloud NAT, and Cloud Storage. This setup provides a solid base for running a data platform, including components like a GitHub Runner and Prefect Worker. The process of setting up these additional components will be covered in the next article of this series, building upon the foundation we’ve established here.

Our dedicated repository contains all code examples and implementations discussed in this article, which can be accessed for reference and further exploration. We encourage you to review the repository for a comprehensive understanding of the concepts presented.

Remember, while this guide provides a strong starting point, it’s crucial to continually assess and adjust your infrastructure to meet your organization’s changing needs and to stay aligned with best practices in cloud computing and data management.

Organizing Networking for Data Platforms: Key Connectivity Options

2025-03-05T09:50:00Z

A poorly designed network can cripple even the most advanced data platform. Slow queries, failed data transfers, and security vulnerabilities often stem from overlooked networking decisions. Yet, networking remains one of the least understood aspects of data architecture.

The Extract, Load, and Transform (ELT) process has become the standard for data integration. It enables organizations to move raw data from source systems to destinations like data warehouses, where it can be analyzed using Business Intelligence (BI) tools. While many aspects of this process deserve attention, networking is a critical yet often underestimated component.

Building a data platform that supports ELT processes requires a clear understanding of how all components communicate. Whether implementing an on-premise solution with open-source tools, leveraging cloud providers, or utilizing SaaS or PaaS solutions, the common thread is the need for seamless connectivity between all elements.

In this article, we’ll explore the options of organizing networking in data platforms, covering key connectivity options, security considerations, and best practices. To lay the groundwork for our discussion, let’s first examine the optimal organization of a data platform.

Data Platform Architecture and Networking

The diagram below presents the reference architecture for the ELT process as a whole, outlining the key components and workflows involved. Each stage has its own phases, with Ingest being part of data extraction, Land being the process of loading data, and Prepare with Model being the transformation.

To better understand how networking ties into a data platform, let’s examine a second diagram, which shifts focus to the networking aspects of the data platform architecture.

The diagram illustrates various components of a data platform, each requiring network configuration to ensure smooth and secure data movement. At the core of our setup is workflow orchestration, which manages the data integration process. Tools like Prefect, Airflow, or Azure Data Factory can handle this, running data flows across various stages.

Extract

The initial phase of the ELT (Extract, Load, Transform) process is data extraction. Every data platform needs to gather data from external systems, represented as “data sources" in the diagram. To access resources from a private environment, we need to use a gateway that allows us to reach external resources. This could be:

Internet Gateway - For accessing public resources.
NAT Gateway - Allow resources in private subnets to connect to services outside the private network.
VPN Gateway - Establishes a secure tunnel with private resources within a different network of our organization or a partner.

Load

Once extracted, data needs to be loaded into a central repository—typically a Data Warehouse. This can be hosted within the same network as the workflow orchestration tool or exist as an external resource.

Same Network: Configuration is simpler as the same team is likely responsible for setting up both components along with networking.
External Resource: Requires additional networking considerations, but the same principles apply—ensuring secure, reliable connectivity.

Transform

The Transform phase follows a similar working pattern, as the workflow orchestration tool needs access to the Data Warehouse. The same resources need to communicate with each other, regardless of whether it’s the Load or Transform phase.

Data Consumption

The final stage is data consumption, where users and tools query the Data Warehouse. Given that sensitive information such as Client Identifying Data (CID) may be stored, secure connections are essential. BI tools, accessible by data platform consumers, need controlled access to the Data Warehouse. Such tools are often managed by a different team from those responsible for data gathering, loading, and transformation.

Application Layer Security

When discussing the networking aspects of data platforms, it’s essential to understand the context within the OSI (Open Systems Interconnection) model. This article primarily focuses on the Network Layer (Layer 3) and Transport Layer (Layer 4)—the backbone of data connectivity. These layers handle IP addressing, routing, and basic connection establishment, forming the foundation for gateways and other networking components.

However, security doesn’t stop at the network level. The Application Layer (Layer 7) plays a critical role in securing data and applications. While this article centers on network infrastructure, robust Layer 7 security is just as important. Common security measures include:

OAuth for secure authorization
Mutual TLS (mTLS) for encrypted, authenticated communication
Basic authentication for simple access control
API gateways for managing and securing API access
Web Application Firewalls (WAF) for protecting against application-level attacks

Regardless of network configuration, Application Layer security should always be implemented. A key principle to remember is that the weaker the Layer 7 security measures, the stronger the network-level controls must be to compensate. This inverse relationship between application-level and network-level security is key to maintaining overall system integrity.

That’s why, while our focus remains on network infrastructure, a holistic approach to data platform security should consider all relevant OSI layers, especially when dealing with sensitive data and critical business intelligence tools.

With this foundation in place, let’s dive into the specific networking options available for securing your data platform.

Connectivity Options

While there are many possible approaches to networking configurations, we’ll focus on the most common scenarios applicable to the majority of data platform use cases:

Public access
Public access with Access Control List (ACL)
VPC peering (within a single project and multiple ones)
Site-to-site VPN

We’ll explore each in detail, followed by a brief discussion of additional networking possibilities.

Public Access

Public Access is the least secure networking option, as it does not restrict access at the network level. Resources residing in a private network are configured to access the internet, while the target resource has no network security applied. This doesn’t necessarily mean the resource is available to anyone, as Application Layer security may still be in place. However, from a networking perspective, access is unrestricted.

This configuration exposes resources to potential attacks, as malicious actors can easily reach them and attempt to bypass application security. Whenever possible, such unrestricted access should be avoided.

That said, Public Access remains the best option for specific, low-risk data sources, such as:

Exchange rates for currencies
Stock market prices
Other publicly accessible data needed in ELT pipelines

To mitigate risks associated with Public Access, organizations can implement additional security measures within their private networks. For instance, they can configure a firewall to block access to all public resources except those explicitly whitelisted. This approach adheres to the principle of least privilege, ensuring only necessary connections are allowed.

By implementing such measures, organizations can balance the need for access to public data sources with maintaining a secure network environment.

Public Access with Access Control List (ACL)

When a system is publicly available, in addition to securing access through Application Layer security controls, we can implement networking mechanisms to expose the system only to a limited group of servers or users. An Access Control List, often referred to as a whitelist, is a security mechanism implemented on the publicly available target system. While the simplest scenario involves allowing access for specific IP addresses, ACLs offer more sophisticated options, including:

Source and destination IP addresses
Port numbers
Network protocols (e.g., TCP, UDP, ICMP)
Time ranges for when the ACL is active

For ACLs to be effective, the system must have a fixed IP address. If this cannot be guaranteed, more secure alternatives like Site-to-Site VPN should be considered.

ACLs can be implemented at multiple levels, including routers, firewalls, or other network devices, providing a layered approach to security. Additionally, ACLs can be used for both inbound and outbound traffic, allowing for fine-grained control over data flow in both directions.

However, while they enhance security, they should not be relied upon as the sole protection mechanism—they work best alongside other security measures, such as authentication, encryption, and regular security audits.

VPC Peering

For cloud environments, VPC (Virtual Private Cloud) Peering allows direct, private network connections between different cloud resources without exposing traffic to the public internet

Since cloud providers use different naming conventions (AWS: accounts, Azure: subscriptions, GCP: projects), we’ll use the term “project” to refer to these organizational units.

VPC peering should be considered the default network configuration for resources within the same cloud provider, regardless of the specific implementation option.

The implementation process varies depending on whether the VPCs are located within the same project or separate ones. Therefore, we will discuss these scenarios separately to highlight their unique characteristics and requirements.

VPC Peering Within a Single Project

Connecting two networks within the same project is a streamlined process. It requires no additional permissions and can be configured entirely from a single account. This peering effectively extends the network, making all resources in the peered network accessible from the first network.

As with other networking options, additional firewall rules or ACLs can be implemented to restrict access directionality or limit connectivity to specific services.

VPC Peering Across Separate Projects

When peering VPCs between different projects, additional security and administrative steps are required:

Cross-project peering permissions must be explicitly granted.
Approval is needed from administrators in both projects.
Firewall rules must be configured in each project to enable cross-project traffic.

Despite these additional requirements, VPC Peering remains the most secure and efficient method for connecting resources within the same cloud provider, offering greater control and reduced exposure compared to internet-based connections.

Site-to-site (S2S) VPN

Site-to-Site VPN is a secure networking solution that connects two or more separate networks, typically in different physical locations, enabling them to communicate as if they were directly connected. This technology creates an encrypted tunnel over the public internet, allowing organizations to securely link their geographically dispersed offices, data centers, or cloud resources.

Key aspects of Site-to-Site VPN:

VPN Gateways: Specialized devices or software applications are deployed at each network endpoint to act as tunnel terminators.
Encryption: Data is encrypted before entering the VPN tunnel and decrypted upon reaching its destination, ensuring confidentiality during transit.
Tunneling Protocols: Protocols like IPsec establish the secure tunnel and manage encryption/decryption processes.
Routing Configuration: Network administrators configure routing to direct traffic through the VPN tunnel instead of the public internet.

A Site-to-Site VPN is one of the most secure ways to connect resources across different locations. While the tunnel relies on public internet infrastructure, all traffic is encrypted, ensuring that data cannot be decrypted without the secret key used to establish the connection. Because of this, securely sharing the secret key is crucial and should never be transmitted through unencrypted channels. With strong encryption and secure key management, Site-to-Site VPN provides an excellent solution for organizations requiring high levels of data protection and privacy across geographically dispersed networks.

Other Networking Possibilities

There are more advanced options available for securing network traffic and isolating it from the public internet. Two notable solutions worth mentioning are:

MPLS (Multiprotocol Label Switching)

MPLS is a packet forwarding technology that operates between Layer 2 and Layer 3 of the OSI model. It typically utilizes a dedicated network infrastructure, ensuring no public connection is involved. Implementing MPLS requires finding a vendor capable of leasing physical cables for exclusive use. While more expensive and complex to implement than previously mentioned options, MPLS offers enhanced security and guaranteed connection speeds.

Dedicated Link

Cloud providers offer solutions like Google Cloud’s Dedicated Interconnect or AWS Direct Connect, which are faster to implement than MPLS, as the cloud provider handles much of the infrastructure. These options are ideal for establishing physical, private connections between on-premises networks and cloud provider networks. However, they may be excessive for connecting to a single data source on a data platform.

While these options provide additional layers of security and performance, they should be carefully considered based on specific organizational needs and resources.

Conclusion

Selecting the right networking strategy for your data platform is critical to ensuring security, performance, and scalability. From public access to VPC peering and site-to-site VPNs, the choice of networking strategy significantly impacts your data platform’s security, performance, and flexibility. Each option comes with trade-offs that need to be considered.

Security should be a primary concern. Public access is the least secure option, while site-to-site VPN offers robust protection.
VPC peering provides an excellent balance of performance and security for resources within the same cloud provider.
Access Control Lists (ACLs) offer an additional layer of security for public access scenarios, allowing for fine-grained control.
Application layer security remains crucial regardless of the chosen networking option, complementing network-level protections.

When designing your data platform’s networking architecture, consider your specific use case, security needs, and scalability requirements. Remember that a comprehensive approach, combining appropriate networking strategies with robust application-level security measures, will provide the most effective protection for your valuable data assets.

As technology evolves, staying updated on best practices and emerging solutions will help ensure your platform remains secure and efficient in the long run.

What is a Modular Data Platform?

2025-02-10T09:30:00Z

Modern data analytics can get complicated. With an abundance of tools, conflicting methodologies, and ever-evolving technologies, mistakes can be costly. However, at its core, data analytics remains grounded in a few fundamental principles. Understanding these fundamentals while leveraging modular and well-designed data platforms can significantly improve operational efficiency and decision-making.

History of Data Platforms: From OLAP Cubes to Hadoop to Lakehouses

The evolution of data platforms has been driven by two primary goals:

Doing Analytics Better: improving analytics work with more efficient storage and retrieval of business and machine data; moving insights generation closer to the domain experts by improving self-service tools and processes
Doing Better Analytics: increasing the value of analytics by having more and deeper insights; leverage statistical modelling, machine learning and AI to improve the quality of business decisions

And so, while SQL, which was invented in 1975, remains at the core of analytics, there have been significant advances in technology.

OLAP cubes emerged in 1993, introducing multi-dimensional analysis. In the early 2000s, Hadoop revolutionized big data processing, allowing distributed storage and computing. More recently, the Lakehouse paradigm has sought to unify the best aspects of data warehouses and data lakes, improving performance, governance, and flexibility.

Reference Architecture of a Modern and Modular Data Platform

A modern data platform consists of several key components, each playing a crucial role in the data lifecycle. These components enable efficient data movement, transformation, and consumption while ensuring modularity and scalability.

Data Sources

Data sources are the origin of information within an organization. These range from structured databases, APIs, and SaaS applications to unstructured sources such as logs, IoT streams, and social media feeds. Some sources offer modern APIs for easy integration, while others, particularly legacy systems, require extensive workarounds.

Ingestion

Ingestion refers to the process of transferring data from sources into the platform reliably. This is typically done via scheduled batch jobs, though some architectures incorporate real-time ingestion using event brokers like Kafka.

The ingestion landscape is fragmented, with numerous tools available, such as Azure Data Factory and Fivetran. However, no tool provides connectors for every possible source. Consequently, organizations often need to develop custom connectors, leading to maintenance challenges and dependencies on vendors.

Landing

Landing zones serve as the initial storage layer where raw, unprocessed data is deposited after ingestion. This stage ensures that data is captured in its original form, preserving fidelity and enabling downstream transformation.

Storage for this type of data (including lakehouse data) has been standardized around the AWS S3 object storage API. Consequently, most cloud providers now offer their own variations of object storage with APIs closely mirroring AWS S3.

Preparation

Since raw data can be messy and inconsistent, preparation is necessary to clean, standardize, and format it for further processing. This stage includes:

Data masking
Data anonymization
Structuring into standardized formats such as Delta Tables or Apache Iceberg Parquet files

Note that both data masking and anonymization could be done also during landing on data “in-transit” to avoid storing sensitive information on the platform.

Data engineers typically handle this step using workflow tools like Alteryx, Azure Data Factory, or programming languages such as Python.

Modeling

Modeling transforms prepared data into well-structured datasets optimized for analytical use. Historically, this was the “T” in ETL (Extract, Transform, Load). Today, tools like dbt have popularized the concept of modular and scalable transformation workflows.

Consumption

Once modeled, data is consumed in various ways, including:

Traditional dashboards and Excel reports
Embedded analytics within applications
AI-powered data exploration (e.g., generative AI and natural language querying)

Data Cataloging

A data catalog is a comprehensive inventory of an organization’s data assets, documenting their structure, relationships, and usage. It extends beyond datasets to include analytical assets such as dashboards, reports, and Jupyter notebooks, ensuring a unified and well-organized view of available information.

Despite its critical role in data management, data cataloging is often overlooked or deprioritized in analytics projects. However, a well-maintained data catalog is fundamental to effective data governance and security. By systematically identifying all data assets, their ownership, and their respective domains, organizations can enhance discoverability, streamline compliance efforts, and facilitate data democratization.

Data Orchestration

Data orchestration refers to the automated coordination of ETL processes, from data ingestion and preparation to final modeling for consumption. It ensures that data flows seamlessly across different stages, reducing manual intervention and improving efficiency.

This industry is highly fragmented, with traditional IT approaches relying on UI-based tools such as Talend and Azure Data Factory. More modern methodologies, however, focus on code-driven orchestration using tools like Apache Airflow and Prefect. These newer solutions provide greater flexibility, scalability, and integration capabilities, making them preferred choices for organizations aiming to build robust and automated data pipelines.

Understanding Modularity in the Context of Data Platforms

Unlike ERP systems, which are often monolithic, data platforms are inherently modular. The diversity of data workflows and use cases makes it impractical to consolidate everything into a single tool.

Some vendors, such as Databricks and Microsoft Fabric, attempt to provide an all-in-one solution. However, even these platforms require integration with external components to cover all aspects of data management.

Components and Interfaces Before Tools

The success of a data platform hinges on well-defined interfaces between its components. A common pitfall is over-reliance on a single vendor, leading to inflexible architectures that struggle to adapt to evolving business needs. Organizations should prioritize:

Well-defined interfaces between tools
Single point for managing accesses (i.e. using A/D groups)
Loose coupling between components to enable flexibility

Workflows and Developer Experience

A streamlined developer experience is crucial for maintaining data platform efficiency. Poorly designed workflows can introduce bottlenecks, reduce productivity, and increase technical debt. Best practices include:

Automating repetitive tasks (e.g., CI/CD for data pipelines)
Enforcing coding standards and documentation
Providing self-service capabilities for data consumers

Data Governance and Security

With numerous tools and evolving datasets, data governance and security must be proactive rather than reactive. Traditional IT governance models, which assume static datasets, are insufficient for modern data platforms. Without a structured approach to creating and managing new data assets, governance becomes impossible.

Effective data governance requires:

Clear company policies on data access, privacy, and security.
A solid understanding of analytics workflows to incorporate governance steps and audit reviews seamlessly.
Automated data cataloging to maintain visibility into data assets and their ownership.
A comprehensive inventory of all analytics tools, ensuring each one is correctly configured and continuously monitored for compliance.

While data governance is straightforward in principle, it requires a structured, realistic approach with well-defined steps to ensure its successful implementation and long-term effectiveness.

Conclusion

Modern data platforms are modular ecosystems that require careful design and governance to be effective. By understanding the historical evolution of data architectures, organizations can make informed decisions about structuring their platforms. Prioritizing interoperability, developer experience, and security ensures a scalable and efficient data operations strategy.

Organizations that embrace modularity and best practices in data management will not only improve operational efficiency but also gain a competitive advantage in an increasingly data-driven world.

Breaking Down Prefect Deployments To Improve The Data Ops Efficiency

2025-01-28T09:45:00Z

When building data platforms, it’s tempting to focus entirely on the technology stack—choosing shiny tools, debating between bulk loads or streaming, and designing storage and infrastructure to meet current needs. Yet, the rush to get data flowing often overshadows a crucial question: How will we monitor and operate all of this effectively?

In the early stages, data projects typically start small: an MVP, one or two data sources, and a couple of flow runs per day. At this scale, operations often feel secondary— issues can be solved on the spot, and data engineering teams are under pressure to deliver data to the end users. But as the platform scales, this oversight catches up. Within months, many teams find themselves struggling to manage DataOps, with operational gaps threatening their progress.

Observability and day-to-day functionality are the bedrock of robust, scalable, and maintainable data pipelines. Modern orchestration tools like Prefect excel at breaking down pipelines into smaller, more manageable pieces, making it easier to monitor, troubleshoot, and deploy smoothly. By designing pipelines with intention and visibility in mind, teams can ensure their data platform remains reliable—even as it evolves.

Why Observability Matters in ETL Processes

Observability is a cornerstone of modern data engineering and operations. As ETL pipelines become critical for decision-making, data teams need deep visibility into pipeline performance and meaningful, actionable logs. The stakes are high—when something goes wrong, time is lost (and as we all know, time is money, or at least that is what they say), and teams are left scrambling to identify issues. At best, this means tedious log analysis and guesswork; at worst—handling complaints from frustrated end-users.

To avoid these pitfalls, observability is a must. It not only ensures transparency with stakeholders but also equips teams to diagnose and address problems efficiently. Effective observability hinges on four dimensions:

Transparency: Understand what each step in the pipeline does, including inputs and outputs.
Traceability: Track data as it flows through the pipeline, making it possible to pinpoint where issues arise.
Granularity: Drill down to isolate performance bottlenecks, failed tasks, or long-running tasks.
Scalability: Expand monitoring and alerting systems to keep pace as the ETL process grows in complexity.

The Pitfalls of a Single Monolithic Flow

When starting an ELT project, it’s common to build one or two monolithic flows. These flows often contain dozens of tasks, which can inevitably grow as the solution scales.

The code usually looks then more or less like this:

1. Task to fetch a list of tables from MS SQL

@task
def get_table_names(conn_str: str) -> List[str]:
    """
    Connect to an MS SQL database and return a list of tables.
    """
    query = """
    SELECT TABLE_NAME
    FROM INFORMATION_SCHEMA.TABLES
    WHERE TABLE_TYPE = 'BASE TABLE'
      AND TABLE_CATALOG = DB_NAME()
    """
    with pyodbc.connect(conn_str) as conn:
        cursor = conn.cursor()
        cursor.execute(query)
        results = cursor.fetchall()

    table_names = [row[0] for row in results]
    return table_names

2. Task to extract data from a specific table into a DataFrame

@task
def extract_table_to_df(conn_str: str, table_name: str) -> pd.DataFrame:
    """
    Run SELECT * on the given table and return a Pandas DataFrame.
    """
    query = f"SELECT * FROM {table_name}"
    with pyodbc.connect(conn_str) as conn:
        df = pd.read_sql(query, conn)
    return df

3. Task to write a DataFrame to S3 as a Parquet file

@task
def write_parquet_to_s3(df: pd.DataFrame, bucket: str, table_name: str):
    """
    Write the given DataFrame as a Parquet file to the specified S3 bucket.
    """

    s3_path = f"s3://{bucket}/{table_name}.parquet"

    df.to_parquet(
        path=s3_path,
        engine="pyarrow",
        index=False,
        storage_options={
            "key": get_secret_from_gcsm("AWS_ACCESS_KEY_ID"),     
            "secret": get_secret_from_gcsm("AWS_SECRET_ACCESS_KEY")},
    )

    return s3_path

4. Main Flow orchestrating the above tasks

@flow
def ms_sql_to_s3_flow(
    conn_str: str,
    bucket: str,
):
    """
    A Prefect flow that loads all tables from MS SQL into S3 as Parquet files.
    """
    # Fetch all table names
    tables = get_table_names(conn_str)

    # For each table, extract and load
    for table_name in tables:
        df = extract_table_to_df(conn_str, table_name)
        write_parquet_to_s3(df, bucket, table_name)

At first, this approach might seem efficient. A single flow can ingest all objects from a database in one run—straightforward and convenient, right?

Initially, with just 10 objects in the database, it works well enough. But as the source database grows to 100 or more items, the cracks begin to show. Usually, this approach introduces several significant challenges:

Difficult Monitoring: A single failure makes the entire flow as failed, forcing data engineers to dig through logs to identify the problematic element.

Limited Reusability: It’s hard to run deployments for one table or only failed objects without re-running the entire flow.
Reduced Scheduling Flexibility: Monoflow might require running all tasks together, even when only a subset of tasks needs frequent execution.
SLA Reporting: Measuring success rates becomes much harder. Reporting on flow run states is unreliable since the failure on one table out of 1,000 causes the whole flow to be marked as failed. Again, this requires digging into logs to measure performance accurately.
Execution Time: Monolith flows are time-consuming and don’t allow parallel execution, hindering efficiency.

In essence, a monolithic approach limits observability, reduces performance, and complicates operations.

The Case for Granulated, Focused Flows

When it comes to sizing your ELT flows, trust me—you’d rather fight 100 duck-sized horses than one horse-sized duck. In other words, breaking down monolithic flows into smaller, focused units is the key to scaling effectively.

The first step is modularizing the monolithic flow. Ideally, each deployment flow should represent a single data object. For example, if you’re ingesting data from an SQL database, think about organizing your process to allow for per-table scalability—it might require more time investment but will divide the complexity.

With the right tools, this approach is not as complex as it sounds. Prefect allows defining deployments with YAML, leveraging project-level default configurations stored under the definitions: key in the prefect.yml file. There are two main ways of using them:

using the entire value as-is,
using part of the pre-defined values (eg. overriding only a single parameter).

This way, you can stick with the pre-defined daily schedule as it is, which makes the deployment creation way easier than it initially seemed.

Here’s why granular flow deployments are worth the effort:

Parallelism: Each table flow can run independently in parallel with others. If one table experiences performance degradation, it does not immediately affect the rest. And yes, it can be included in the monoflow, but why spend time reinventing the wheel? Orchestrator can take care of that.
Monitoring and Error Handling: If a single table fails, its flow run alone fails. This allows one to quickly identify the failed table, debug it, and restart only that deployment. Also, it helps with monitoring the execution time of a particular table or with tracking data quality issues.
Improved Data Quality Testing: It’s much easier to enable data quality tests per data object instead of having universal rules. Is it better to have customized tests per column in the data set or check if the set is not null only?
Incremental Maintenance and Scalability: Modular flows create clear boundaries. Adding or updating flows for new or modified tables doesn’t necessarily affect existing deployments. Each table’s logic is easier to maintain and evolve in isolation.
Version Control: Each deployment can be versioned independently. This makes testing changes for one table more straightforward and also makes the CI/CD implementation easier.
Team Collaboration: Different engineers can own specific deployments, making it easier to distribute responsibility and keep changes localized. It’s good to use tags to identify project-related deployments—e.g., it’s possible to have a sales tag in Prefect for sales data-related processes.
Granular Scheduling: Some tables need to be refreshed three times daily, but some should be reloaded monthly only. The granular approach allows for more playing with the schedule.
SLA Reporting: It’s simpler, as the real situation is shown on the run level, and failure means real failure.

Conclusion

In conclusion, a granular approach to orchestrated deployments is more than just a technical choice—it’s a strategic advantage. By breaking large, monolithic pipelines into focused, modular flows, data teams gain clearer observability, easier troubleshooting, and the flexibility to handle diverse scheduling needs

Focusing on key concerns—performance, reliability, and maintainability—can help you build a better data solution using a granular approach. Over time, this approach will lead to more predictable, scalable, and maintainable ETL processes.

dlt and Prefect, a Great Combo for Streamlined Data Ingestion Pipelines

2025-01-27T12:54:00Z

Doing data ingestion right is hard…

Despite advances in data engineering, data ingestion, which includes the Extract and Load (EL) steps of the ELT process, remains a persistent challenge for many data teams.

This complexity is often due to the real-world limitations of open-source tools, leading teams to opt for UI-based solutions. While these tools are great for getting started quickly, they often lack the flexibility and scalability required for production-grade data platforms.
In the era of AI, UI-based tools face one more limitation: they miss out on most of the benefits of the advanced code generation capacity of modern LLMs (Large Language Models)[1].

Even if teams do decide to use open-source solutions, they often end up creating volumes of low-quality glue code. This in-house software, typically written in a rush by non-professional engineers, often fails to meet essential requirements for modern data platforms, such as EaC (Everything as Code), security, monitoring & alerting, reliability, or extensibility. Moreover, since it’s written by non-professional engineers, such code is far more brittle and much harder to maintain and modify. Consequently, all modifications to the code (such as adding new features or fixing bugs) take much more time and are far riskier than they should be.

…but there is light at the end of the tunnel

Luckily, in recent years, with the growing adoption of software engineering practices, we’ve seen a professionalization of the data engineering field. This has resulted in the creation of a number of high-quality, open-source tools that simplify and improve the quality of data engineering work, such as dlt and Prefect.

In this article, we explore how dlt and Prefect can be seamlessly integrated to implement a best-practice data ingestion component of a modern data platform. Our insights are grounded in real-world experience designing and implementing scalable, code-based data platforms with these open-source tools.

A Short Introduction to dlt and Prefect

dlt

dlt is a Python data ingestion framework enabling data engineers to define connectors and pipelines as code. It offers a rich set of features for building best-practice pipelines and supports both built-in and custom connectors built with regular Python code.

dlt ingests data in three stages: extract, normalize, and load. The extract stage downloads source data to disk. The normalize stage applies light transformations to the data, such as column renaming or datetime parsing. The load stage loads the data into the destination system.

Here’s a compact guide to key dlt concepts:

dlt config: dlt can be configured in three ways: through files (config.toml and secrets.toml), environment variables, and Python code.
Using config.toml for default settings is recommended, as it’s easy to store the file together with pipeline code on git. While it can contain some pipeline-level settings as well, its main purpose is to configure global behavior such as logging, parallelization, execution settings, and source or destination configuration common to all pipelines.
Resource and Source:
A resource is a representation of a single item in a dataset. It can be a file, a database table, a REST API endpoint, etc.
A source is a collection of resources, such as a filesystem (eg. s3), a database, or a REST API.
By applying hints to the resource with resource.apply_hints(), we can configure extraction settings specific to the resource, a pipeline, or a pipeline run, such as primary key, cursor column, column typing, partitioning, etc. We can also apply some light transformations to the data (eg. data masking) before it’s loaded to the destination with the resource.add_map() method.
dlt is flexible when it comes to working with sources and resources, and it’s easy to use either, depending on the need.
Pipeline: In dlt, pipeline describes the flow of data from a source (or resource) to a destination. Each pipeline handles a single source<->destination pair and takes a source or resource as input.
Pipelines can be reused to ingest different resources each run. For example, we can have one “Postgres to S3” pipeline, but ingest each Postgres table separately due to different scheduling or configuration needs.
A pipeline definition contains pipeline- or pipeline run-specific destination configuration, as well as settings for the load phase of the ingestion. Under the hood, a pipeline run (pipeline.run()) executes each pipeline step: extract (pipeline.extract()), normalize (pipeline.normalize()), and load (pipeline.load()).

Prefect

Prefect is a Python data orchestration library that allows data and machine learning engineers to define data workflows (data ingestion, transformation, model training, etc.) as code. It provides a rich set of features to help engineers implement best-practice data orchestration workflows.

Its cloud offering eliminates the historically stressful and labor-intensive maintenance of data orchestration systems.

Let’s unpack the core concepts of Prefect:

Task: A task is a single unit of work in a Prefect flow. It describes a single step to be executed in the workflow.
While it’s possible to implement the logic of the step directly in the task, in most cases, we recommend keeping tasks as thin wrappers around regular Python functions.
Flow: A flow is a collection of tasks that define a data workflow. You can think of it as a graph of tasks, describing their relationship (eg. this task should always run after this one, and this other task should run after that one, but only if it fails).
Similar to a dlt pipeline, the same flow can be reused with different sets of parameters. An instance of a flow with specific parameter values is called a deployment.
In this article, we utilize this fact by utilizing a single extract_and_load() flow capable of executing any dlt pipeline, depending on the parameters passed to it. As a result, each ingestion becomes a new Prefect deployment rather than a new flow, which has a major consequence: deployments can be defined with YAML, which means that they don’t require any Python code to be written, which means users don’t need to set up a local Python development environment just to eg. ingest a new table with an existing pipeline. Instead, we can, for example, expose a simple application that allows non-technical users to create new deployments with a few clicks.
Deployment: A deployment is a way to run a flow with a specific set of parameters and environment configuration. While most environment configurations in Prefect would typically be defined at the workspace level, deployments allow for overriding some of these settings, including on a per-run basis, which simplifies testing and debugging.

Creating Data Connectors and Pipelines with dlt

Now that we’ve covered the theoretical underpinnings of dlt and Prefect, it’s time to see these concepts in action. We’ll explore how to implement best-practice dlt pipelines and bring these tools to life.

Data Pipeline Features

Alright, before we dive into the technical part, let’s start with the basics. A production-grade data pipeline needs to have several key features:

Modularity: The pipeline should be designed to allow the reuse of components across multiple pipelines.
Extensibility: The pipeline must be upgradeable without disrupting ongoing production jobs.
Reliability: The ability to inspect pipeline execution and quickly identify and resolve issues is crucial.
Security: Proper mechanisms must be in place to securely store and access secrets.
Privacy: Data storage should adhere to privacy regulations, ensuring compliance.
Efficiency: Pipelines must be optimized for cost-effective execution.

Data pipelines aren’t one-size-fits-all, and achieving a production-grade pipeline involves ensuring those key features. But how to get there?

Modularity

To achieve modularity, it’s best to split the dlt pipeline code into the following structure:

├── pipelines
│   ├── a_to_c.py
│   ├── b_to_c.py
│   └── utils.py

In this structure, a_to_c.py and b_to_c.py represent two example pipelines, each handling data from a source system (a and b) to a destination system ©.

The utils.py file contains common utilities such as data masking implementation, default configuration for source and destination systems, or default pipeline configuration (except configuration specified in dlt’s config.toml; for more information, see the dlt config paragraph in the dlt section).

Extensibility

Implementing extensibility goes beyond modularity. The code should also be testable, and ideally, automated testing should be integrated into the CI/CD process.

Since dlt pipelines are implemented using Python, they can be tested with common tools like pytest. Unit tests should focus on custom utility functions, while integration tests verify the entire pipeline’s behavior.

For integration testing, use a local database or disk drive instead of the target database. DuckDB is a great choice for this purpose, as it’s a lightweight, in-memory database that can be used to inspect the loaded data quickly.

Reliability

To maintain trust with data platform users, make sure that when production pipelines fail, you are informed immediately and can recover quickly. While we recommend implementing alerting in the orchestration layer, pipeline recoverability depends on having access to detailed logs.

Luckily, dlt provides rich built-in logging and error-handling mechanisms. It’s a good idea to also enable progress monitoring for additional useful information, such as CPU and memory usage.

Security

dlt supports various ways of storing credentials. For local use, secrets can be stored in a .dlt/secrets.toml file, while production environments may benefit from an external credential store, such as Google Cloud Secret Manager. To accomplish this, you can store the secret retrieval utility function in utils.py and reuse it within your pipelines.

However, since we’re using Prefect for orchestration, it’s also possible to follow a different path and use Prefect Secrets to store the credentials.

Privacy

Data anonymization and/or pseudonymization are crucial to ensure compliance with privacy regulations. Data can be erased/anonymized either:

During the ingestion phase (in which case the original data never enters the destination system)

During the transformation phase (in which case private data is stored in one or more layers in the destination system but hidden from the eyes of end users)

While dlt doesn’t provide built-in anonymization features, it does provide the necessary tools to implement the first option effectively.

For more information, see the example in the official documentation.

Efficiency

To ensure pipelines are both cost-effective and high-performing, several optimization techniques can be applied:

Incremental extraction
Loading data incrementally allows for reducing the amount of data that needs to be extracted. Currently, dlt supports incremental extraction for its core sources: REST API, SQL database, and filesystem.
Incremental extraction allows us to download only new or modified data.
Write dispositions
Write dispositions work in tandem with the two extraction methods to reduce the amount of data that needs to be loaded. For example, if you only extracted new and modified data, you don’t want to overwrite existing data, as that would result in data loss. In such a case, insert the new records and update the existing ones instead.
Parallelization
dlt allows parallelizing each stage of the pipeline utilizing multithreading and multiprocessing (depending on the stage).
In cases where further parallelization is needed (i.e., the workload exceeds the capacity of a single machine), utilizing orchestrator-layer parallelization may be required. However, this scenario is now rare, as large virtual machines capable of processing petabytes of data are widely available, and dlt can leverage the machine’s resources more efficiently than older tools or typical in-house Python code.
Various other optimizations

As the topic of incremental loading can be complex even for seasoned data engineers, we’ve prepared a diagram of all the viable ELT patterns:

NOTE: dlt also provides sub-types of the “merge” disposition, including SCD type 2; however, for clarity, we did not include these in the diagram. For more information on these subtypes, see relevant documentation.

The choice of a specific implementation depends on what is supported by the source and destination systems as well as on how the source data is generated. Ideally, incremental extract should be used whenever possible. Then, whether you choose the “append” or “merge” write disposition depends on how the data is generated: if you can guarantee that only new records are produced and no existing data is ever modified, you can safely use the “append” disposition. Next, you need to check if the destination system handles the disposition you intend to use (eg. some systems don’t support the “merge” disposition).

The following diagram from dlt’s official documentation also provides a good overview of when to choose which write disposition:

Orchestrating Data Pipelines with Prefect

Orchestrating data pipelines with Prefect can streamline your workflow and significantly improve efficiency. Let’s dive into the best practices for implementing Prefect flows and how they integrate smoothly with your data pipelines.

Orchestration Job Features

Ideally, the orchestration layer is a thin wrapper over the underlying data pipeline logic. Whenever a feature can be implemented at the pipeline level, it should be implemented there in order to prevent excessive coupling with the orchestration layer and minimize complexity, which simplifies self-service data ingestion.

Here are a few key features that are best handled at the orchestration layer:

alerting
additional reliability measures
security (specifically, secret management)
distributed processing

Alerting

With Prefect, you can set up alerts, ensuring you’re notified via Slack, Teams, or email whenever jobs or infrastructure components enter an unexpected state.

Reliability

While we can (and, where possible, should) implement retries and timeouts at the pipeline level, Prefect provides these features at the task and flow level. Think of this as a last-resort, catch-all mechanism that allows data engineers to ensure timeouts and retries are enforced regardless of how well the dlt pipeline or helper code is written, again lowering the bar for self-service data ingestion.

Secret Management

Security is always a top concern, and Prefect’s secret management integrations make it easier than ever to store and handle secrets. Whether it’s Google Cloud Secret Manager or AWS Secret Manager, Prefect allows you to securely retrieve credentials and pass them to the dlt pipeline. This approach ensures that no credentials are stored locally, and administrators have fine-grained control over access by utilizing Prefect’s Role-Based Access Control (RBAC).

Distributed Processing

While any code-based orchestration tool allows for distributed processing, this feature is rarely required at the pipeline level in recent times. Firstly, data ingestion tools such as dlt are capable of efficiently utilizing machine resources, including parallelization and efficient and safe use of memory. Secondly, virtual machines have grown bigger—we can now easily rent VMs with hundreds of cores and hundreds of gigabytes of RAM. Therefore, typically, distributed processing is only required in case we need to run multiple resource-hungry pipelines in parallel.

Production Workflow

Now that we’ve outlined the essential features of a production-grade dlt pipeline and Prefect flow, let’s break down the steps of creating and orchestrating data ingestion pipelines in production.

Overview

The diagram below illustrates the key steps in this production workflow.

Create a dlt pipeline: We start by creating a dlt pipeline (if the one we need doesn’t exist yet). Once the pipeline is finished and tests pass, we can move on to the next step.
Create Prefect deployment: We create a Prefect deployment for the pipeline. Notice we utilize Prefect’s prefect.yaml file together with a single extract_and_load() flow capable of executing any dlt pipeline to drastically simplify this process.
Create a Pull Request: We create a pull request with the new deployment. This triggers the CI/CD process.
DEV environment: The deployment is created in the DEV Prefect workspace, and a DEV Docker image is built. We can now manually run the deployment in Prefect UI, which will execute our pipeline in the DEV environment.
PROD environment: Once we’re happy with the results, we merge the pull request. This triggers a CI/CD job, which creates the deployment in the PROD Prefect workspace and builds a PROD Docker image. The deployment schedule is also only enabled at this stage.

If the pipeline already exists and only a new table is being ingested, the user needs only add a few lines of YAML toprefect.yaml and create a PR.

Configuring dlt

While dlt is highly configurable and allows for a lot of customization and optimization, we recommend starting with three highly useful configurations:

runtime.log_level to enable more logging
normalize.parquet_normalizer.add_dlt_load_id to add a dlt load ID to the loaded data
normalize.parquet_normalizer.add_dlt_id to add a unique id to each row.

The ID settings will make our data easier to work with for downstream users, as well as make our loads (especially incremental ones) easier to debug.

Creating a dlt Pipeline

Pipeline Design

We start by creating a dlt pipeline, following the best practices detailed in the Creating data connectors and pipelines with dlt section above.

For testability and modularity, we recommend splitting the pipeline into a resource (source data) and pipeline (journey and destination) parts. This way, you can easily test each part separately.

Inspecting the Data Manually

At any stage of pipeline development, you can manually inspect the loaded data, e.g., by printing it to the console or by checking the database directly.

Testing the Pipeline

For integration testing, you can use DuckDB as a destination system. It’s lightweight and allows you to quickly check ingested data, so you can iterate faster.

Creating a Prefect Flow and Deployment

Flow Design

After the dlt pipeline is working, it’s time to wrap it in a Prefect task and flow. Keep the orchestration layer simple—use a single extract_and_load() flow for all data ingestion tasks. With Prefect deployments handling the pipeline name and arguments, you can set everything up with just a few lines of YAML.

Handling Pipeline Secrets

Secrets should be passed through a special dictionary parameter, such as secrets. These secrets should then extracted from Prefect blocks and forwarded to the dlt pipeline, ensuring they are securely handled.

Deploying to Production

A pull request with the new deployment should automatically trigger the CI/CD process in our project repository’s CI/CD pipelines. We will soon dive deeper into how to implement this process using GitHub Actions in a separate article, so stay tuned!

Summary

Building a modern, scalable data platform starts with mastering data ingestion, which requires tools that are as powerful as they are flexible. By combining dlt for efficient, open-source data pipelines with Prefect for orchestration, you can create workflows that are not only production-ready but also streamlined for both developers and data teams.

This approach ensures flexibility, scalability, and cost-effectiveness, making it ideal for modern data platforms while also strategically positioning your platform to excel in the upcoming AI age.

Next steps

Data Transformation

dlt and Prefect (with the help of dbt) are just as good at data transformation as they are at data ingestion. Stay tuned as we explore how to integrate these tools for data transformation in a future article!

Ready to Dive Deeper?

If you’re ready to build a cutting-edge data platform with dlt and Prefect, get in touch. We offer expert guidance to help you set up every component and provide a fully equipped template Git repository with production-grade code. No fluff—just practical, scalable solutions designed to handle real-world challenges and set your data workflows up for long-term success.

Footnotes

[1] While more and more UI-based tools add copilot capabilities, they face several fundamental limitations:

Copilots, while text-based, are limited by the UI tools they are built upon.

Imagine instructing someone to build a complex LEGO castle with only a basic set of blocks. No matter how clearly you explain, the result will always be limited, forcing you to find workarounds.

These UI tools often use a custom language to define data pipelines, which adds another layer of complexity.

As the quality of LLMs is highly reliant on the size and quality of the dataset they’re learning from, it means these assistants cannot reach the same level of fluency as LLMs trained on much more popular languages, such as Python.

Imagine the person you’re instructing to build your LEGO castle has very little experience with LEGO or construction in general. They would struggle to understand basic jargon and construction trade practices, and they would often make mistakes requiring your intervention.

Deploying Prefect on any Cloud Using a Single Virtual Machine

2025-01-15T09:29:00Z

Choosing the right data platform architecture is quite a challenge for any organization. It’s a balancing act: you need something that delivers immediate value while staying flexible enough for future growth, all without sacrificing scalability, simplicity, or efficiency.

This article offers a thoughtful guide to the decision-making process behind choosing Prefect with lightweight Kubernetes (K3S) on a single Virtual Machine (VM) with any cloud provider. You’ll explore:

Why simplicity and flexibility are essential for modern data platforms.
Key considerations for selecting the right data orchestration tool.
Insights into serverless vs server-based execution of Prefect flows.
Approaches to running a server-based Prefect worker

Rather than a step-by-step tutorial, this guide is designed to help you make solid platform architecture decisions and design a solution tailored to your organization’s unique needs. Let’s dive in.

Challenges With Picking a Data Platform Architecture

The options for building a data platform are endless, but many fall short. With the rise of affordable cloud storage, expectations have changed, leaving many once-revolutionary legacy systems struggling to keep up. At the same time, new solutions making big claims often fail, either missing critical features or bogging organizations down with unnecessary complexity. For smaller companies, the challenge is even greater—a data platform should drive business value, not require a dedicated team just to maintain it.

Starting small may seem practical, but early shortcuts can turn into major obstacles as the platform grows. Undoing poor architectural choices later is often costly and disruptive. That’s why choosing a solution that is both simple and scalable from the outset is essential.

For decision-makers, this journey begins by stepping back and evaluating both the current state of their team and the platform they rely on. The Data Platform Maturity Curve is a helpful framework for this:

Depending on the organization’s data technology maturity level, your platform must adapt. This article focuses on those in the middle of the curve, where simple scripts and ad-hoc solutions are no longer enough, but advanced features like autoscaling aren’t yet necessary. At this stage, the platform delivers tangible business value and is steadily becoming integral to operations. Downtime—whether it lasts hours, a day, or even a week—is growing increasingly expensive.

The goal? A platform that’s lightweight, scalable, and future-ready without overcomplicating things.

Data Platform Orchestration: The Key to Seamless Integration

Even the best-designed data platform is useless if it’s not integrated. No matter how carefully you choose your architecture, your platform’s success hinges on how well its core components—ingestion, transformation, and serving—work together. These phases can only operate efficiently when they are tightly aligned.

Adapted from “Fundamentals of Data Engineering: Plan and Build Robust Data Systems” by Joe Reis & Matt Housley

Early-stage platforms often rely on manual orchestration, which works at first but quickly becomes a bottleneck as data grows and workflows become more complex. Managing, ensuring accuracy, and reducing downtime requires a more structured approach.

A few basic improvements can help push the boundaries further. For instance:

Instead of running all scripts locally, they can be executed on a virtual machine.
Setting up a database helps centralize data
Basic automation of workflows can be managed with cron jobs in Linux.

While these incremental improvements help in the short term, significant challenges remain:

Manual code execution becomes increasingly error-prone as the scale of operations grows.
Cron jobs become difficult to manage as workflows become more complex and interdependent. Debugging failures can quickly turn into a nightmare, especially with cascading issues across multiple flows.

This is where automated data orchestration becomes the key to streamlining workflows across the entire lifecycle. It allows teams to automate, monitor, and scale operations by transforming disconnected processes into a cohesive system, minimizing manual intervention and reducing errors.

Let’s review the most popular options available in the market.

Data Orchestration Tools

The three leading orchestration tools in the market are:

Apache Airflow: An open-source and community-driven tool with robust features but a steep learning curve. Managed versions like Google Cloud Composer and Amazon MWAA simplify deployment but tie users to specific cloud providers.
Prefect: A modern, cloud-agnostic, and easy-to-configure solution emphasizing scalability, portability, and developer-friendly features that allow for flexible orchestration. Prefect’s architecture also supports running workflows in hybrid environments, seamlessly bridging on-premises and cloud solutions.
Dagster: Designed for data-aware orchestration, Dagster prioritizes validation, lineage, and developer productivity, making it ideal for teams handling complex pipelines.

At The Scalable Way, we have worked with both Airflow and Prefect in a few projects. We advise Prefect for a lightweight setup with fewer deployment things to worry about.

What is Prefect Cloud?

Prefect Cloud is a fully managed orchestration platform that simplifies running and monitoring Python-based workflows without the overhead of managing infrastructure. It’s well-suited for teams looking to automate data workflows, from ingestion and transformation to serving.

Its strengths include:

Scalability: Handles thousands of workflows with ease.
Monitoring and alerting: Built-in features simplify issue detection and resolution.
Cloud-agnostic architecture: Runs seamlessly across environments, avoiding vendor lock-in.

By automating the orchestration layer, Prefect Cloud allows teams to focus on building robust pipelines without the overhead of managing infrastructure.

Common Struggle for Prefect Users: Deployment

Adopting Prefect as an orchestrator unlocks many possibilities, but like any powerful tool, it comes with a learning curve. Prefect flexibility and a developer-first approach can initially feel daunting for teams unfamiliar with building solid deployment solutions.

Prefect’s philosophy emphasizes providing tools rather than prescribing solutions, allowing users to adapt its features to their specific needs. While this approach offers flexibility and scalability, it can leave data engineers uncertain about where to start with scalable deployment practices like CI/CD pipelines and autoscaling.

Eternal Dilemma: Server-based or Serverless

Another consideration is choosing the right setup for running Prefect flows. There are two primary approaches, each designed to cater to different needs:

Server-based: This requires setting up infrastructure such as virtual machines, lightweight Kubernetes (e.g., K3S), or managed Kubernetes clusters. While these setups provide maximum control, scalability, and adaptability, they demand a higher level of expertise and upfront effort.
Serverless: Managed solutions like Prefect Cloud’s service or serverless compute options from cloud providers (AWS Fargate, Google Cloud Run, Azure Container Instances) eliminate the need for infrastructure management, making them appealing for simpler workflows.

Serverless solutions, though convenient, are best suited for simpler workflows, as they come with five notable challenges:

Startup Overhead: Prefect Worker images often have heavy dependencies, increasing flow initialization time. This leads to latency, as serverless platforms can introduce delays between task executions due to event-driven triggers. A long-running server with a persistent Prefect Worker is usually much quicker.
Vendor Lock-In: Serverless solutions are often tightly integrated with specific cloud providers, making it difficult to migrate workflows across platforms. Even Prefect Work Pools, though useful, have limited functionality at the Pro tier.
Cost Management: Serverless can be cost-effective for intermittent workloads, but can become expensive with unpredictable usage patterns. Managing costs is trickier compared to traditional server-based setups.
Limited Control and Security Concerns: Serverless architectures limit control over the execution environment, as all logic runs on cloud provider-managed machines. This raises security risks, especially for companies dealing with sensitive data or operating in highly regulated industries, due to reduced visibility and potential vulnerabilities in shared infrastructure.
Token Management and Data Access Risks: Serverless setups require Prefect to hold a token for accessing cloud resources, creating security risks if mismanaged. Server-based setups mitigate this by reversing the data flow, allowing the server to pull from Prefect, and reducing the risk of data breaches or unintended data exposure.

Ultimately, the choice between server-based and serverless depends on the teams’ needs and stage of data maturity. However, for most organizations aiming to scale, a Prefect Work Pool running on a long-running server is a more optimal and reliable solution.

Deployment Options for a Server-based Data Platform

Local Prefect Worker Process

Connects directly to Prefect Cloud and serves as an introductory setup to understand Prefect Cloud’s functionality. However, this is not suitable for production scenarios due to limited scalability and resilience.

Systemd Process on Single or Multiple VMs

Runs Prefect flows in Docker containers, providing a lightweight setup that is relatively easy to configure. This approach is well-suited to small projects and teams, as Docker limits unnecessary complexity.

Single VM with Lightweight Kubernetes (K3S)

It’s not as simple as a Systemd setup because of the introduction of Kubernetes and Helm. Thanks to these tools, it’s more scalable and adaptable for future growth. This setup offers flexibility for migration to more robust configurations as project demands increase.

Managed Kubernetes Cluster

The most feature-rich solution-managed Kubernetes supports autoscaling, spot instances, and integrations with tools like Active Directory. It is ideal for comprehensive data platforms. However, this approach adds operational complexity and may be excessive for smaller projects.

Recommended Setup for Getting Started: Lightweight Kubernetes on a Single Virtual Machine

The lightweight Kubernetes on a single Virtual Machine (VM) setup strikes an ideal balance between cost efficiency and operational flexibility. By leveraging lightweight Kubernetes (K3S), you gain the core benefits of Kubernetes with significantly reduced overhead, making it perfect for smaller environments or projects with constrained resources. Its streamlined architecture ensures smooth operations without the complexity of managing a full Kubernetes cluster. The diagram illustrates a basic architecture that effectively meets most requirements for running Prefect flows in a scalable manner.

Using Helm charts to deploy the Prefect Worker simplifies orchestration, ensuring seamless integration with existing systems while minimizing manual configurations. Helm also makes updates easier, promotes standardization, and reduces deployment errors.

Running everything on a single virtual machine keeps the infrastructure simple yet scalable. If project demands grow, you can easily upgrade the VM or expand to a multi-node cluster without major changes to your architecture. Additionally, this setup simplifies maintenance, provides clear monitoring and debugging paths, and avoids vendor lock-in, preserving flexibility for future enhancements.

Conclusion

Building a modern data platform is no easy task. Success lies in keeping it simple while ensuring flexibility and scalability. With the right tools and setup, like Prefect and lightweight Kubernetes on a single virtual machine, you can create a platform that delivers immediate value and adapts as your needs grow.

By focusing on scalable, modular solutions, you’re not just solving today’s problems—you’re building a platform ready for whatever comes next.