As one of the world's leading media and entertainment companies, NBCUniversal relies on data to deliver unparalleled entertainment experiences across its global broadcast, cable, and streaming networks. With content delivery demanding real-time analytics, complex data pipelines, and advanced machine learning, the company's data infrastructure is a critical business asset that supports decision making across the portfolio.
NBCUniversal has been at the forefront of innovation for 100 years and recognized a need to modernize its data infrastructure to support its growing content ecosystem while maintaining operating costs. As data volumes grew and analytics demands evolved, the company sought a more scalable and integrated solution. The existing infrastructure supported core analytics needs, but also highlighted opportunities to enhance flexibility, improve performance for advanced workloads, and optimize compute expenses to strengthen operational efficiency.
“NBCUniversal has been on a journey to operationalize and broaden access to data for teams across the company, this required a new solution that allowed us to work faster,” said Kevin Hill, Senior Vice President, Data Technology and Product, NBCUniversal. “Databricks provided us with a way to spend less time building and more time analyzing the data.”
When NBCUniversal set out to modernize its data platform, the goal was to unlock the agility to scale around major content launches, awards shows, and live events without overprovisioning. Partnering with EXL and Databricks, NBCUniversal migrated onto Databricks' job-specific compute, unifying analytics and ML on a single platform and bringing 300 analysts along for the ride.
Results at a glance
30% reduction in data infrastructure costs via job-specific compute
300+ analysts onboarded smoothly onto Databricks SQL
Elastic scaling for high-traffic events without pre-reserved slot waste
Unified ML + analytics through MLflow and Lakeflow Jobs
The challenge: balancing cost and performance
As NBCUniversal continued to innovate, data became even more critical to decision making within organizations across teams like marketing, product, and content. The existing architecture could no longer sufficiently manage the amount of data or queries needed to support the rapidly evolving organization. A slot-based reservation model allowed the organization to reserve numerous slots to support these parallel workloads, but resulted in significant compute costs and resource contention.
“As today’s content ecosystem rapidly evolves, our partners across the company need to make informed decisions faster,” said Louie Kuznia, Vice President, Data Engineering, NBCUniversal. “Databricks provided a solution that could support us in modernizing NBCUniversal’s data infrastructure without compromising agility or speed.”
Evolving cost considerations, along with the desire for greater flexibility for advanced use cases such as machine learning and enhanced governance around data lineage, motivated NBCUniversal to explore alternative platforms.
The solution: Databricks Lakehouse Architecture
NBCUniversal partnered with Databricks and EXL, a global consultancy, to execute a strategic migration to the Databricks Data Intelligence Platform. This partnership combined EXL's deep migration expertise with Databricks’ leading data solutions to create a modernized data infrastructure supporting data engineering, machine learning, and advanced analytics.
The decision to work with Databricks offered several advantages, including enhanced flexibility and overall efficiency for data workloads:
Unified analytics platform: Databricks provides a single platform for data engineering, data science, machine learning, and business intelligence workloads, which eliminates the need to maintain multiple specialized tools.
Flexible lakehouse architecture: Databricks leverages DeltaLake and Apache Iceberg™ with open source Apache Spark™ and Lakehouse (Databricks SQL), enabling flexible handling of both structured and unstructured data across diverse formats.
Advanced analytics capabilities: Databricks excels in supporting complex analytics, including machine learning workloads, with native integration of MLflow for experiment tracking and model versioning.
Granular compute control: Databricks’ architecture enables more granular control over compute resources, with each data pipeline running on dedicated compute rather than competing for shared slots.
The migration strategy: a phased, partner-led approach
The migration to Databricks requires transformations to data structures, modifications to code, adjustments to data formats, and careful orchestration of upstream and downstream dependencies. EXL and Databricks together developed a comprehensive, phased approach that minimized operational disruption while accelerating the transition.
Phase 1: Assessing the existing landscape
The team began with comprehensive discovery, creating an inventory of existing workloads, tables, and codebases while thoroughly analyzing their dependencies. This assessment phase identified the artifacts requiring migration, organized them by priority, and established success criteria.
Phase 2: Designing the target state architecture
The design phase focused on mapping existing constructs to their Databricks equivalents while optimizing for cost and performance.
Key design decisions included:
Workspace architecture: Structuring workspaces according to the Databricks Well-Architected Framework, mapping its best-practice pillars to NBCUniversal’s organizational structure and security requirements to ensure a scalable, secure, and future-proof environment.
Data governance strategy: Implementing Unity Catalog as the central governance layer.
Compute strategy: Defining job cluster configurations, instance types, and autoscaling policies tailored to each workload's requirements.
Storage layout: Designing Delta Lake table structures with appropriate partitioning, clustering, and predictive optimization to automatically maintain efficient file layouts, statistics, and storage for sustained performance at scale
Phase 3: Proving the concept with custom migration tools
Before committing to a full-scale migration, EXL executed a targeted MVP using a representative subset of NBCU's workloads. This pilot phase served two purposes: to validate the migration approach and to develop custom automation tools that would accelerate the broader effort.
During the MVP, EXL built four proprietary accelerators specifically for NBCU's environment:
SQL translation engine: An intelligent code converter that analyzes SQL syntax and automatically transforms it into optimized Spark SQL, handling dialect differences, function mappings, and performance tuning.
Orchestration migrator: A tool that parsed existing DAGs and regenerated them as Lakeflow Jobs, preserving dependencies and scheduling logic.
Data transfer orchestrator: An automated pipeline that moved data to Delta Lake, selecting optimal transfer methods based on table size and update frequency.
Validation framework: A comprehensive testing suite that compared query results, row counts, and aggregate statistics between source and target systems.
These accelerators transformed what would have been months of manual work into automated, repeatable processes with built-in quality checks.
Phase 4: Migrating in strategic waves
With accelerators validated and the target architecture finalized, the team executed the migration in carefully sequenced waves. Rather than migrating all workloads simultaneously, they adopted a risk-based approach that prioritized high-value, lower-complexity pipelines first.
Each migration wave followed a consistent pattern:
- Workload selection: Identify a cohesive group of related tables and pipelines for migration.
- Automated code generation: Use the SQL Translation Engine to convert queries and the Orchestration Migrator to rebuild workflows.
- Data synchronization: Transfer historical data while maintaining incremental updates from existing infrastructure.
- Quality assurance: Run the Validation Framework to ensure data accuracy and completeness.
This iterative approach allowed the team to maintain business continuity and refine their processes with each wave, applying lessons learned to subsequent migrations.
Phase 5: Validating through dual operations
The final phase before cutting over to Databricks involved running both platforms in parallel for an extended validation period. For several weeks, NBCUniversal operated dual pipelines, executing the same transformations in both the existing platform and Databricks while comparing the outputs.
This parallel operation served multiple critical functions:
- Production validation: Ensure parity with Databricks-generated outputs under real-world conditions with production data volumes.
- Performance benchmarking: Measure actual execution times and costs in the Databricks environment versus historical performance.
- Stakeholder confidence: Provide business users and downstream consumers with empirical evidence that the migration preserved data quality and availability.
- Cutover planning: Identify any edge cases or corner scenarios that require additional attention before fully decommissioning legacy platforms.
Only after achieving consistent results across all validation metrics did NBCUniversal proceed with the final cutover, retiring existing workloads pipeline by pipeline with full confidence in the new platform.
Business impact: 30% cost reduction and strategic gains
The results of NBCUniversal's migration were substantial and measurable:
Compute cost reduction: “By transitioning to Databricks’ job-specific compute, NBCUniversal achieved a 30% reduction in overall data infrastructure costs. Each pipeline now runs on its own dedicated compute resources, enabling independent scaling during peak workloads with automatic scale-down during low-demand periods.”--Ludwig Kuznia, Vice President, Data Engineering at NBCUniversal
Improved operational flexibility: The parallel execution model provides greater agility in responding to business demands. During known high-traffic events (such as major content launches or awards shows), NBCUniversal can dynamically scale up job cluster capacity without affecting other workloads or incurring the waste of pre-reserved but underutilized slots.
Unified analytics capabilities: Beyond cost, the migration unlocked new capabilities. NBCUniversal's data engineering teams gained access to unified ML operations through MLflow, enabling:
Experiment tracking and model versioning
Streamlined ML lifecycle from training to production
Automated workflow standardization through Lakeflow Jobs
Enhanced data culture: The migration catalyzed broader organizational change. By simplifying infrastructure management and reducing operational overhead, NBCUniversal created conditions for a more agile, collaborative, and data-driven culture between the data engineering and the analyst teams. Teams could focus on business problems rather than infrastructure constraints.
Ease of analyst onboarding: The migration enabled approximately 300 analysts to transition quickly and comfortably to Databricks, thanks to Databricks SQL’s full ANSI SQL compliance and an intuitive platform that streamlines query writing, dashboard creation, and analysis within a unified workspace.
Conclusion: from cost center to strategic asset
NBCUniversal's migration to Databricks demonstrates how strategic technology decisions can transform data infrastructure, unlocking greater efficiency and competitive advantage. By partnering with EXL and embracing Databricks’ lakehouse architecture, NBCUniversal not only lowered its run costs but also unlocked a more agile, unified platform that allows teams to experiment faster, bring new data products to market sooner, and respond more quickly to changing audience and advertiser needs.
The new platform gives NBCUniversal the ability to rapidly iterate on audience insights, optimize ad products in near real time, and support new monetization models that were difficult or too expensive to run before. This combination of lower costs, faster decision cycles, and a single environment for data, analytics, and ML turns the data platform into a differentiator, which enables NBCUniversal to move more quickly than competitors in how it understands, serves, and grows its audiences.
Get started
Ready to modernize your data infrastructure and unlock similar cost savings and analytics capabilities?
Connect with Databricks:
Speak about your use case with Databricks Delivery Solutions Architects (DSAs).
Throughout this migration, the DSA for NBCUniversal worked hand-in-hand with EXL to craft a strategy aligned to the customer's priorities. DSAs accelerate implementation, surface roadblocks early, and coordinate across teams — from assessment and architecture through execution and validation — providing the technical oversight needed to reduce risk and speed delivery. Connect with your Databricks account team to learn how a DSA can extend your migration team.
Visit EXL’s website to learn more about their suite of solutions, including EXLdata.ai.
Explore partner-led migration options to accelerate your journey
Start with an assessment of your current environment
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.