The word "AI" appears in every data platform pitch now. Most of the time, it means nothing — a chatbot bolted onto a dashboard, an LLM that generates SQL, or a marketing slide that says "AI-powered" next to a cron job.

This article is about something different: what happens when you put actual intelligence — machine learning, adaptive optimization, closed-loop feedback systems — at the core of how a data lakehouse operates. Not AI as a feature. AI as the operating model.

The result is a lakehouse that understands its own workloads, optimizes its own storage layouts, routes its own queries, heals its own tables, and continuously improves — without human intervention. That is what an AI data lake actually is. Everything else is automation with better branding.

Table of Contents

Automation Is Not Intelligence

Most data lake "automation" today works like this: a cron job runs compaction every night at 2 AM. It merges small files into bigger ones. It does not know if the table needs compaction. It does not know which partitions are degraded. It does not know what columns queries filter on. It does not know whether last night's compaction actually helped.

That is automation. It follows a fixed rule, regardless of context.

Intelligence is different. Intelligence observes, learns, decides, acts, measures the outcome, and adjusts. A system that compacts a table because its file health crossed a threshold, sorts by the columns that real queries actually filter on, sequences the operation after snapshot expiration to avoid wasted work, and then verifies that query latency improved — that is intelligent.

The distinction matters because data lakes are not static systems. Workloads shift. New dashboards appear. AI agents start querying tables that were designed for nightly batch. Streaming pipelines change write patterns. A schema evolves.

Fixed rules cannot keep up. Intelligence can.

The AI Control Plane

An AI data lake needs a brain — a system that continuously ingests telemetry from every table, every engine, and every query, and uses that signal to drive optimization decisions across the entire lake.

LakeOps is that system. It is an autonomous control plane for Apache Iceberg that puts AI at the center of lakehouse operations — compaction, maintenance, query routing, observability, governance, and agent enablement, all driven by continuous learning rather than static rules.

What makes it a control plane and not another tool: LakeOps is not an engine, not a catalog, and not storage. It connects to whatever you already run — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper, or S3 Tables as catalogs; Trino, Spark, Snowflake, Athena, DuckDB, Flink as engines; S3, GCS, or ADLS as storage — and adds the intelligence layer on top. No data movement. No code changes. No vendor lock-in. Setup takes about ten minutes.

What makes it AI-native: every decision LakeOps makes is informed by learned behavior, not hardcoded rules. It observes query patterns across all engines, predicts table degradation from structural signals, adapts data layouts as workloads shift, learns which engine performs best for each query shape, and measures the outcome of every optimization to improve the next one. It also exposes a native MCP (Model Context Protocol) server so AI agents can query the lake directly — with schema discovery, intelligent routing, and layered guardrails built in. The more the lake is used — by humans and agents alike — the smarter the control plane gets.

Three operating modes let teams choose how much autonomy the AI gets:

  • Autopilot — the AI decides and executes. Define policies and thresholds; the system handles everything.
  • Recommendations — the AI analyzes and recommends with full context (what, why, expected impact). A human approves.
  • Policy-bounded — the AI operates within declared boundaries, escalating edge cases.

Most teams start with recommendations to build trust, then move tables to autopilot as confidence grows. The intelligence is the same in all modes — the only difference is who clicks "execute."

With that foundation, here is how AI transforms each layer of lakehouse operations.

How AI Transforms Each Layer of the Lakehouse

When we talk about an "AI data lake," we are talking about specific capabilities that require learning, prediction, and adaptation — not features that could be implemented with a shell script.

Layer What AI Does What Scripts Cannot
Compaction Learns query patterns across engines, sorts data by the columns actually used in WHERE/JOIN/GROUP BY, adapts as patterns change Static sort keys, fixed file-size targets, blind to query patterns
Maintenance Predicts which tables will degrade next, sequences operations by dependency, prioritizes by business impact Fixed schedules, independent jobs, no awareness of health trajectory
Query routing Learns which engine performs best for each query shape, uses LLMs to reason about new templates, adapts as tables improve Static rules, manual engine assignment, no cost optimization
Observability Correlates structural signals with query performance, predicts degradation before user impact Threshold alerts, no predictive capability, no cross-signal correlation
Governance Adapts policy enforcement to workload patterns, recommends changes from observed behavior Static policies, manual updates, no feedback from operations
Agent integration Feeds agent telemetry back into optimization, shapes the lake around AI access patterns No awareness of agents, no closed-loop optimization

Each of these deserves a deeper look.

AI-Driven Compaction: Learning How Data Is Read

Traditional compaction is a janitor: it sweeps up small files and merges them. It does not know why the files exist or how they are read. An intelligent compaction engine does.

Query Pattern Analysis

LakeOps continuously observes every query that hits every table — across Trino, Spark, Snowflake, Athena, DuckDB, Flink, and any other connected engine. It extracts which columns appear in WHERE, JOIN, and GROUP BY clauses. It tracks how these patterns change over time. It weights recent queries more heavily than old ones.

This is not a one-time analysis. It is a continuous learning loop.

From this analysis, the AI knows — per table — which columns matter most for data layout. A table where 80% of queries filter on event_date and region should have its data physically sorted by those columns. A table where queries mostly join on customer_id should sort differently.

From Analysis to Action

During compaction, data files are physically re-sorted by the learned columns. The result: Parquet row-group min/max statistics become effective, and engines skip irrelevant data without reading it. Predicate pushdown and column pruning actually work — not because someone manually configured sort keys, but because the AI figured out what the right sort order is.

On production tables, this AI-driven sorting reduces data scanned by 51% and delivers 12x faster queries compared to unsorted compaction. The numbers are large because the difference between "sort by whatever was configured at table creation" and "sort by what queries actually need" is enormous.

Adaptive Sort Evolution

Here is where intelligence really separates from automation. When query patterns shift — a new dashboard starts filtering on product_category instead of region, a data science team changes their join keys, AI agents start accessing different columns — the sort strategy evolves on the next compaction pass. No configuration change. No ticket. No human intervention.

A table whose workload stabilizes gets compacted less frequently — the AI recognizes that the current layout is already optimal and avoids redundant work. A table whose workload spikes gets prioritized. This is resource allocation driven by learned behavior, not fixed schedules.

Speed as an AI Enabler

The AI's compaction decisions need an engine fast enough to execute them continuously — not queue them for a nightly batch window. LakeOps runs compaction on a purpose-built Rust engine powered by Apache DataFusion: 221 seconds where Spark takes 1,612 seconds (95% faster), at roughly $5/TB versus $50/TB (90% cheaper). Peak throughput reaches 2,522 MB/s. Tables that OOM Spark — a 1.2 TB table with 12,000 files, for example — complete without special configuration because memory is bounded, not heap-dependent.

That speed is not just a performance metric. It is the enabler for AI-driven compaction: when compaction takes minutes instead of hours, the system can respond to workload changes on the same day they happen rather than falling behind. The AI can iterate. Fast execution makes continuous learning practical.

AI-Driven Maintenance: Predicting What Breaks

Iceberg tables require six maintenance operations: data compaction, snapshot expiration, manifest consolidation, orphan cleanup, position delete resolution, and statistics computation. Every platform team knows this. The question is when and in what order to run them — and that is where AI changes the game.

Health Prediction, Not Health Reaction

Traditional monitoring tells you a table is degraded after queries slow down. By then, users have already filed tickets and data pipelines are late.

LakeOps predicts degradation before it hits. The AI continuously scores every table on structural signals — file count trajectory, partition balance trends, manifest growth rate, snapshot accumulation velocity, delete-file ratios. It sees the pattern: "this table's small-file ratio is climbing at 4% per day; at current trajectory it will hit Critical in 72 hours."

This is a classification and prediction problem that humans cannot solve manually at scale. A platform team with 800 tables across multiple catalogs cannot eyeball metadata trends for each one. An AI system that ingests every commit, every query, and every structural change can.

Intelligent Sequencing

The order of maintenance operations matters. Running orphan cleanup before snapshot expiration risks deleting referenced files. Running manifest rewrites before compaction wastes the rewrite since compaction will invalidate file references.

The AI learns the correct dependency graph and sequences operations per table:

  1. Compaction first (merge small files, apply learned sort order)
  2. Snapshot expiration after (remove old references safely, respecting active readers)
  3. Manifest consolidation next (consolidate after file references stabilize)
  4. Orphan cleanup last (reclaim newly unreferenced storage)

This is not just a fixed ordering — the AI evaluates whether each step is needed based on current table state. A table with healthy manifests skips manifest consolidation. A table with no orphans skips cleanup. The maintenance plan is tailored per table, per cycle, based on real signals.

Operation Outcome Learning

After each operation completes, the AI measures the outcome: did compaction actually reduce query latency? Did snapshot expiration free the expected storage? Did manifest consolidation improve planning time?

These outcomes feed back into future decisions. If compacting a particular table's hot partition consistently delivers a 5x query speedup, the system learns to prioritize that partition. If expiring snapshots on a rarely-queried table has minimal impact, it deprioritizes that work. Over time, the maintenance engine gets better at deciding what to do — not just executing a predefined list.

AI-Driven Query Routing: Matching Queries to Engines

Most production lakehouses use multiple query engines. Trino for interactive analytics. Spark for heavy ETL. Snowflake for BI dashboards. DuckDB for ad-hoc exploration. Athena for pay-per-query workloads. The problem is deciding which query goes where.

Without intelligence, this is a manual decision — or worse, no decision at all, and every query hits the same engine regardless of whether it is a 50 MB selective filter or a 500 GB full table scan.

The Three-Router AI Stack

LakeOps implements intelligent routing through QueryFlux, an open-source Rust-based SQL proxy, extended with a three-layer AI stack:

1. Adaptive routing (learned behavior). The AI tracks historical execution data for every query template — which engine ran it, how long it took, how much it cost, whether it succeeded. For previously seen query shapes, routing is instant: the system already knows which engine delivers the best cost-performance ratio. Cached decisions take 0ms.

2. LLM-based routing (reasoning). For new query templates the system has not seen before, an LLM reasons over the query structure combined with live table statistics — file count, data size, sort order, partition layout, health score. It predicts which engine is the best fit. This handles the cold-start problem that pure historical routing cannot.

3. Semantic routing (intent matching). When neither historical data nor structural analysis alone is sufficient, semantic matching infers the query's intent — exploratory analysis, dashboard refresh, ETL pipeline, or AI agent retrieval — and routes accordingly.

The three routers compose. Over time, more query templates enter the adaptive cache and routing gets faster and more accurate.

Data-Quality-Aware Routing

This is where AI routing connects to AI compaction and AI observability. The routing layer knows — from LakeOps's health scoring — whether a table is well-compacted, fragmented, or degraded. It adjusts routing decisions accordingly:

  • A well-compacted table with good sort order? Route to a lightweight, cheaper engine that benefits from efficient data skipping.
  • A degraded table with 40,000 small files? Route to an engine that can handle the overhead while compaction runs in the background.
  • A table that was just compacted and is now healthy? Update the routing weights so future queries use cheaper engines.

As the AI compaction engine improves table health, the AI routing engine shifts queries to cheaper engines. The cost savings compound. Production benchmarks show up to 56% workload cost reduction from intelligent routing alone.

AI-Driven Observability: From Dashboards to Diagnosis

Traditional observability is a dashboard: it shows metrics, you interpret them. AI-driven observability is a diagnostic system: it interprets the metrics for you, tells you what is wrong, explains why, and either fixes it or recommends a fix.

Cross-Signal Correlation

LakeOps does not monitor file counts and query latency independently. The AI correlates signals across layers:

  • Storage signals — file count, size distribution, access patterns, orphan volume
  • Metadata signals — manifest count and depth, snapshot velocity, partition skew, statistics coverage
  • Engine signals — query latency, scan volume, CPU utilization, per-engine field access
  • Operation signals — maintenance duration, compaction throughput, health score changes

From these, it produces actionable insights with causal chains. Example: "Query latency on raw_clickstream increased 8x because 312 partitions exceed the file count threshold — scan amplification is 8x. Compaction is recommended and will reduce query time to baseline."

That is not a threshold alert. That is a diagnosis.

Severity-Ranked Insights

The AI surfaces per-table findings at four severity levels — CRITICAL, HIGH, WARNING, LOW — each with the specific metric, current value, threshold violated, and a remediation action. Critical and High Insights trigger automated remediation. Warning and Low Insights surface for human review.

The key AI element: the system does not just check "is file count above X?" It evaluates the combination of signals. A table with many small files but no queries gets a lower severity than a table with fewer small files but high query frequency. Business impact is inferred from workload patterns, not just structural metrics.

Predictive Degradation

Instead of alerting after a table degrades, the AI predicts when it will degrade based on trajectory. A table whose snapshot accumulation rate will hit its retention limit in 48 hours triggers an alert now — with enough lead time to act before users notice.

AI-Driven Governance: Policies That Adapt

Static governance policies are set-and-forget — which usually means set-and-break. Workloads change, new tables appear, query patterns shift, and the policies that made sense six months ago no longer match reality.

Intelligent Policy Recommendations

LakeOps's AI observes actual workload behavior and recommends policy adjustments:

  • "Table orders_archive has not been queried in 90 days. Current retention policy keeps 30 days of snapshots. Recommend reducing to 7 days to reclaim 18 GB of metadata."
  • "Namespace analytics.* consistently produces small files from streaming writes. Recommend adding an auto-compaction policy with a 1-hour trigger window."
  • "Table customer_events is queried by 3 AI agents with PII columns. Recommend adding PIIMaskGuard policy to agent-facing routing groups."

These are not generic suggestions. They are derived from observed behavior — query patterns, write patterns, access patterns, and health trajectories specific to each table.

Policy Inheritance with Workload Awareness

Policies cascade from organization to catalog to namespace to table — but the AI ensures inheritance makes sense. A compaction policy designed for batch tables is not blindly applied to streaming tables in the same namespace. The system flags conflicts between inherited policies and actual workload patterns, and either adjusts automatically (in autopilot mode) or recommends changes (in manual mode).

Policies apply uniformly across catalogs — AWS Glue, Polaris, Nessie, Gravitino, Lakekeeper — without per-catalog scripts or engine-specific logic. Every policy is auditable, versioned, and controllable with a single toggle.

The Closed Loop: How Every Query Makes the Lake Smarter

The most important concept in an AI data lake is the closed-loop feedback system. This is what separates an intelligent system from a collection of automated scripts — and it is the core of how LakeOps operates.

Here is the loop:

    Queries hit the lake
    (engines, agents, dashboards, pipelines)
            │
            ▼
    Telemetry captures patterns
    (columns, engines, latency, cost, frequency)
            │
            ▼
    AI analyzes and learns
    (sort order, routing weights, health scores,
     maintenance priorities, policy fitness)
            │
            ▼
    Optimizer acts
    (compact with new sort, reroute queries,
     adjust policies, trigger maintenance)
            │
            ▼
    Outcomes measured
    (query faster? cost lower? health improved?)
            │
            ▼
    Learnings feed back ──────► next cycle

Enter fullscreen mode Exit fullscreen mode

Every query — whether from a BI dashboard, a Spark ETL job, or an AI agent — feeds signal into this loop. The more the lake is used, the better it gets at organizing itself. This is the defining characteristic of an AI data lake: usage improves performance, rather than degrading it.

In a traditional lake, usage degrades performance — more queries means more load, more writes means more small files, more engines means more fragmentation. In an AI data lake, usage is signal. More queries means better sort-order decisions. More engines means richer routing data. More writes means more opportunities for the optimizer to learn and act.

AI Agents as Lake Citizens

AI agents introduce a fundamentally new workload pattern. They issue SQL iteratively — dozens of queries per reasoning step — at high frequency, without human review, with expectations of sub-second latency. A single customer-support agent answering one ticket can issue 30–50 SQL statements across user history, orders, product metadata, and support logs. Multiply that across hundreds of concurrent agent sessions and the workload profile looks nothing like the BI traffic your lake was designed for.

Read: Routing Multiple Query Engines with Iceberg

The Infrastructure Problem

Agents querying uncompacted tables pay a 5–10x latency penalty. When an agent runs 12 sequential queries in a reasoning loop, each 3-second query delay adds up to 36 seconds of wall-clock time for a single interaction. Users perceive this as the agent being broken. And unlike a human analyst who learns to avoid expensive queries, an agent in a loop repeats the same mistake indefinitely — compounding costs every iteration.

The LakeOps MCP: Agent-Native Connectivity

The Model Context Protocol (MCP) is the emerging standard for how AI agents interact with data infrastructure — filling the same role for agent-to-lakehouse communication that REST catalogs fill for engine-to-catalog communication.

Read: Agentic AI MCP for Iceberg

LakeOps exposes a native MCP server that any compatible agent — Claude, LangChain, LlamaIndex, or custom — can connect to with zero integration code. The MCP server provides four purpose-built tools rather than a generic "run anything" endpoint (agents that get a single execute_sql tool tend to skip schema discovery and hallucinate table names):

  • list_schemas — returns all available schemas and databases across connected engines. Agents call this before constructing queries to avoid guessing at namespace structures.
  • describe_table — returns column names, types, partition specs, sort orders, and optionally sample rows in a single call. Eliminates the multiple round-trips agents typically make to understand a table's shape.
  • execute_query — runs SQL through the full routing and guardrail pipeline, with optional engine_hint and enforced max_rows to prevent unbounded result sets from flooding the LLM context window.
  • explain_query — returns estimated cost, row count, and execution plan without running the query. Agents use this for pre-flight cost checks.

Each routing endpoint gets a stable URL — for example, storefront-analytics.lakeops.dev. Agents connect to that URL and inherit the full policy stack (guardrails, routing rules, cost limits) automatically. No per-agent configuration. Wire compatibility spans PostgreSQL, MySQL, and Arrow Flight SQL, plus async query support with SSE streaming so agents in tool-use loops can handle long-running queries gracefully.

Intelligent Guardrails

Unsupervised agents need enforcement at the query level, not at the application level. LakeOps provides a composable guard chain — every query passes through it regardless of which agent or frontend issued it:

  • ReadOnlyGuard — blocks DDL and DML from agent sessions using SQL parsing (not string matching), catching CTEs that wrap mutations and function calls with side effects.
  • RowLimitGuard — injects LIMIT N when a query lacks one. Agents in reasoning loops often forget to limit results, leading to multi-gigabyte result sets that overflow LLM context.
  • CostEstimateGuard — issues EXPLAIN before execution and rejects queries where estimated scanned bytes exceed a threshold. This is the primary defense against the cartesian-join and missing-WHERE patterns that agents frequently generate.
  • PIIMaskGuard — rewrites queries to exclude, hash, or null-out columns tagged as sensitive. PII never enters the LLM context window.
  • HumanApprovalGuard — pauses high-stakes queries and sends a webhook (Slack, email, or custom) for manual review.

Guards stack per routing group. A typical agent-facing configuration: read_only → row_limit → cost_estimate → pii_mask. A data engineering agent gets: cost_estimate → human_approval for DDL. Every guard action is recorded with full auditability — what fired, what was rewritten, what was rejected.

AI Solving the AI Problem

This is where the AI data lake concept becomes recursive. The same intelligence that optimizes the lake for human workloads also optimizes it for AI workloads — and agents produce particularly rich signal:

  • Higher query frequency — more data points for the adaptive optimizer per unit of time
  • Predictable patterns — roughly 80% of agent queries are repeated templates (same SQL structure, different literal values), making sort-order learning fast
  • Agent context propagation — every query carries structured metadata (agent_id, conversation_id, step_index, tool_call_id) that enables per-agent cost attribution, session replay for debugging, and workload-specific routing optimization

The feedback loop tightens: agents query the lake → telemetry captures their patterns → the AI reshapes data to match → agents get faster results → the agents produce even better signal → the cycle accelerates. Production deployments show agent query p95 latency dropping from 5–10 seconds to under 500ms as tables are compacted and sorted for observed access patterns, with per-query compute cost dropping 65% through intelligent routing.

AI-Native vs. AI-Washed: How to Tell the Difference

Not every product that claims "AI-powered" actually uses intelligence. Here is how to evaluate:

Signal AI-Native AI-Washed
Compaction Learns sort order from cross-engine query patterns; adapts as patterns change Fixed sort keys configured manually; schedule-based execution
Maintenance Predicts degradation; sequences by dependency; learns from outcomes Runs on cron; independent jobs; no feedback loop
Routing Learns cost-performance per query shape; uses LLM for cold-start; adapts weights Static rules; manual engine assignment
Observability Correlates signals across layers; predicts issues; diagnoses root cause Threshold alerts; dashboard without interpretation
Governance Recommends policy changes from observed behavior; adapts inheritance Static policies; manual updates
Feedback loop Closed-loop: every query improves the system Open-loop: no learning from outcomes
Agent support Native MCP with intelligent guardrails and self-optimization REST API bolt-on; no workload-aware optimization

The litmus test: does the system get better over time without human configuration changes? If yes, there is actual intelligence. If no, it is automation with a marketing label.

The Compounding Effect

The individual components are valuable. The compound effect is transformational.

AI-driven compaction reduces data scanned by 51%. That means every query across every engine benefits. Routing then shifts those improved queries to cheaper engines, because well-compacted tables do not need heavyweight compute. Storage costs drop because compaction reduces total file count and orphan cleanup reclaims unreferenced data. Observability improves because the AI has better signal to work with. Agents respond faster, which means fewer retries, lower token costs, and better AI application quality.

Each AI component amplifies the others. Better compaction → better routing → lower cost → more headroom for AI agents → richer signal → better compaction. The loop runs continuously.

Production numbers from LakeOps deployments:

  • 80% reduction in compute and storage costs
  • 12x faster queries after AI-driven compaction and sorting
  • 95% faster compaction than Spark (221s vs. 1,612s on 200 GB)
  • 90% cheaper compaction ($5/TB vs. $50/TB for Spark)
  • 56% workload cost reduction from intelligent routing
  • 100% of tables continuously monitored and maintained
  • ~10 minutes to connect and start optimizing

These numbers compound because the system is a closed loop, not a collection of independent tools.

Where This Is Heading

The trajectory is clear. Data lakehouses are moving from manually operated infrastructure to self-managing systems where intelligence handles the operational layer end-to-end.

Today, LakeOps's AI handles compaction, maintenance sequencing, query routing, health prediction, and policy adaptation. The direction extends to:

  • Automatic schema evolution — the AI detects that a new column is consistently added to queries and recommends schema changes or materialized views
  • Predictive capacity planning — forecasting storage and compute needs based on workload growth trends and seasonal patterns
  • Cross-lake optimization — intelligence that spans multiple lakes, regions, and clouds, optimizing data placement and replication based on global access patterns
  • Agent-native data contracts — AI agents negotiate data freshness, latency, and cost SLAs directly with the control plane, and the system optimizes to meet them

The foundation for all of this is the closed-loop feedback system. A lake that learns from every query is a lake that can evolve to meet requirements that do not yet exist.

Summary

An AI data lake is not a data lake with an AI chatbot. It is a data lake where intelligence drives operations:

  • AI decides how to compact — learning from cross-engine query patterns, not static configuration
  • AI decides when to maintain — predicting degradation, sequencing by dependency, learning from outcomes
  • AI decides where to route — matching query shapes to engines using learned cost-performance models
  • AI diagnoses problems — correlating signals across storage, metadata, engines, and operations
  • AI adapts governance — recommending policy changes based on observed behavior
  • AI optimizes for agents — feeding agent telemetry back into the optimization loop

The closed-loop feedback system is the defining characteristic. Usage improves performance. Every query makes the lake smarter.

LakeOps is the autonomous control plane that implements this intelligence stack for Apache Iceberg — connecting to your existing catalogs, engines, and storage in minutes, with no data movement and no vendor lock-in.

Explore the platform · Read the docs · Get a demo