Data engineering teams often face a “Day 2” operational reality after building a data platform: the ongoing work of maintaining the orchestrator itself.

For the Data Platform team at Checkout.com, managing a self-hosted Apache Airflow environment on another hyperscaler was consuming time the team wanted to spend elsewhere as server management, patching, and incident response were pulling focus from building pipelines.

By migrating to Managed Service for Apache Airflow (Gen 3), Google Cloud’s fully managed Airflow service, Checkout.com transformed its reliability and cost structure. Here’s how they built a more scalable, cost-efficient, and robust data foundation.

The starting point: self-managed Airflow

Before the migration, Checkout.com ran Airflow on self-managed infrastructure. While functional, maintaining the underlying resources required significant attention. Patching, upgrades, and server management created regular interruptions.

Operational data from the past year illustrates some of the challenges the company was navigating:

  • Reducing operational friction: In its self-managed environment, Checkout.com faced stability challenges, particularly during high-load periods. 

  • Complex dependency management: Upgrading packages and ensuring compatibility was a constant, manual struggle. With Managed Airflow (Gen 3), the company was able to simplify this by handling dependencies at the image level, ensuring seamless compatibility out-of-the-box during routine environment upgrades.

  • DAG sync time: Syncing DAGs to the scheduler took approximately six minutes after deployment to S3, which affected iteration speed.

  • Manual processes: Scaling required manual intervention, and onboarding new teams meant manually creating secrets and variables for dbt.

The solution: Managed Service for Apache Airflow (Gen 3)

Checkout.com’s team migrated to Managed Airflow to offload infrastructure responsibility and take advantage of Google Cloud's managed scalability. The results were immediate and measurable across three areas: reliability, cost, and developer velocity.

https://storage.googleapis.com/gweb-cloudblog-publish/images/checkout-managed-airflow-chart.max-2200x2200.png

Dynamic scaling in action

In the previous elastic container service setup, the team allocated the maximum number of workers required for peak loads. This meant paying for peak capacity around the clock, regardless of actual usage.

Managed Airflow provides built-in dynamic scaling, eliminating the need for manual resource management. The environment automatically adjusts the number of workers based specifically on the workload demands. When tasks spike, the system scales up; when they drop, it scales down to save resources. Similarly, moving from fixed provisioning to dynamic scaling reduced monthly costs by an estimated 30%.

Reliability and DAG isolation

Achieving increased stability was a primary driver for Checkout.com’s migration since in the past, a single problematic DAG could affect its entire environment. 

Managed Airflow introduced a number of critical architecture improvements:

  • DAG isolation: Each DAG runs in its own execution environment. If one DAG fails or consumes excessive resources, it doesn’t affect the entire environment.

  • Managed operations: Google Cloud handles patching and upgrades during scheduled windows, removing the need for manual upgrade management.

  • Improved visibility: Integration with Google Cloud Monitoring and Cloud Logging provides clear visibility into task execution. Engineers can now debug issues independently without escalating to the platform team.

Faster developer workflows

The migration also improved day-to-day workflows for Checkout.com’s data engineers.

  • Faster deployments: Using Cloud Storage for DAGs enabled near-instant syncing.
  • Simpler onboarding: Teams no longer needed platform support to create variables before onboarding.
  • Modernizing dbt execution: One of the company’s most significant wins was changing how it runs dbt. Previously, its engineers had to manually install and manage complex virtual environments for every supported dbt version. By leveraging containerized dbt runs, Managed Airflow (Gen 3) eliminates dependency bottlenecks. This ensures complete dependency isolation, allowing teams to run any required dbt model with minimal setup and no manual infrastructure overhead.
  • Environment updates: The company no longer needs to redeploy the entire Airflow environment to add new roles or update Python packages.
  • AI-powered troubleshooting with Gemini Cloud Assist: In a self-managed environment, a failed task often triggered a frantic hunt through fragmented logs and metrics. With Managed Airflow, Checkout.com can initiate a Gemini investigation directly from its Airflow DAG UI in the Google Cloud console. 

Gemini doesn't just provide generic error messages; it generates a scorecard that evaluates different hypotheses with both supporting and contradictory evidence, which can drastically reduce mean time to recovery.

Conclusion

For Checkout.com, the move to Managed Airflow (Gen 3) marked a strategic shift, one that freed its engineers to focus on delivering value.

"With Managed Service for Apache Airflow, we’ve achieved significant improvements in efficiency, scalability, and reliability. Managed infrastructure, automated scaling, faster deployments, and isolated execution environments have transformed how we operate." — Keisi Mancellari, Data Platform Engineer, Checkout.com

With a stable, scalable, and cost-efficient platform in place, Checkout.com is now able to  focus on the future of its data pipelines, confident that its orchestration layer is ready for whatever comes next.

Learn more about Managed Service for Apache Airflow and how it can support your data platform.

Special thanks to the following contributors to this post: Serge Bouschet and Keisi Mancellari

Posted in