The lakehouse community spent this week deciding what belongs in the format and what belongs outside it. Iceberg contributors pushed to retire equality deletes in V4, Parquet voted on a new floating point encoding, and Arrow shipped its 25.0.0 release across every language it supports. Polaris debated the persistence layer that everything else sits on, DataFusion welcomed a new committer alongside a fresh release, and the incubating Ossie project fielded hard questions about maturity and identity. Read together, the dev lists tell a story about a stack that is growing up. The debates are less about whether features exist and more about what guarantees each layer owes the ones above it.

By the numbers, this was a heavy week. Polaris led with 89 messages, Iceberg posted 70, Parquet 65, Ossie 47, Arrow 36, and DataFusion 20. That is 327 messages across the six lists in seven days, spanning release votes, format proposals, persistence design, and governance. Every thread referenced below links to the full conversation on lists.apache.org, so treat this newsletter as a map and go read the primary sources on anything that touches your stack.

Apache Iceberg

The biggest structural conversation on the Iceberg list this week centered on the future of equality deletes. Huaxin Gao revived the long-running proposal to deprecate equality deletes in Iceberg V4, and the thread drew nine messages of substantive engagement. Maximilian Michels gave the streaming perspective that many practitioners will recognize. He called equality deletes the number one pain for streaming use cases and noted that many users give up when they see merge-on-read costs, or they build custom solutions that pull them away from core Iceberg. Michels also shared concrete progress from the Flink side. The index that powers ConvertEqualityDeletes persists in Flink's managed RocksDB state, updates as new data arrives, and checkpoints on a schedule. The conversion works for data written by any engine, but the conversion itself still requires Flink. Storing that index in Iceberg and opening it to all engines is the next step toward an engine-agnostic answer. Michels closed with a +1 for deprecation in V4, and he framed the plan as realistic given the progress since the first conversation in 2024. Watch this one. Removing equality deletes reshapes how every streaming writer targets the format.

Release energy stayed high on the Rust side. The community voted on Iceberg Rust 0.10.0 RC4 across a twelve message thread, passed it, and announced the 0.10.0 release within the week. The Rust implementation keeps shipping at a steady pace, and each release makes it a more credible option for teams that want Iceberg without a JVM. The ecosystem also grew sideways this week. A vote on the Apache Iceberg Terraform provider v0.1.0 RC1 followed an earlier RC0 round, which signals that infrastructure-as-code management of Iceberg resources is close to its first official release. Catalog and table management through Terraform closes a gap that platform teams have filled with custom scripts for years. Think about what the provider unlocks in practice. A namespace, its tables, and their properties get declared in the same repository as the buckets and IAM policies they depend on. Environments get stamped out from the same configuration. Drift between what the catalog holds and what the code declares becomes detectable in a plan step instead of a production surprise. Version 0.1.0 will be small, but the direction matters more than the initial resource coverage.

Two proposals this week asked what Iceberg tables should be allowed to contain. Martin Prammer proposed adding Vortex as an Iceberg file format, and he framed the draft in a smart way. The document splits into two parts. The first part defines the criteria any file format needs to meet to join Iceberg. The second part shows how Vortex meets them. Prammer wants feedback on the criteria before anyone argues about the candidate, because the criteria will outlive this one proposal. Meanwhile, the vector type discussion continued as Philipp Fischbeck backed a dense numeric vector type with non-null elements. He argued for a lean feature list with no vector-specific stats, constraints, or schema evolution in the first pass. He also agreed with Tanmay Rauth that metrics like value count and null count should live at the vector level rather than the element level. Fischbeck pointed at the ongoing Parquet fixed-size list work as the storage foundation, which connects this thread directly to the Parquet discussions covered below. AI workloads are pulling both formats in the same direction at the same time, and the two communities are coordinating rather than duplicating.

Correctness and operations got their share of attention too. Oleksii Omhovytskyi asked about release timing for the encrypted deletion-vector fix, and his message is a model bug report. On a natively encrypted format-v3 table using AWS KMS, a merge-on-read UPDATE or DELETE writes a deletion-vector Puffin file without key metadata. The next read then fails with a null key metadata error. The fix is merged on main with backports to the 1.11.x and 1.10.x branches, and Omhovytskyi verified the 1.11.x build against his exact reproduction. His question is simple: does a 1.11.1 or 1.10.3 patch land soon, or should teams wait for 1.12.0? He offered to test any release candidate. Anyone running encrypted V3 tables with merge-on-read writes should track this thread closely, because the bug blocks reads after routine write operations.

The spec and dependency conversations rounded out the week. Alexandre Dutra opened a discussion on migrating Iceberg to Jackson 3 as frameworks like Spring Boot and Quarkus make the same move. His analysis is candid. The migration is mechanical but pervasive. Artifact coordinates change, package names change, core classes get renamed, ObjectMapper construction moves to a builder pattern, and exceptions become unchecked. Spark and Flink runtimes carry low impact because they already shade Jackson. The real concern sits with downstream users of iceberg-core, because Jackson types leak into the public API of the parser classes, JsonUtil, and the REST HTTP layer. Dutra acknowledged that iceberg-core's public API is permanently Jackson-coupled by design, so the community needs to decide how to sequence a break of that scale.

On the collation front, Alexander Löser and Andrei continued a careful exchange about collation support and the ICU version problem. The core question: how much cross-engine interoperability should the format guarantee versus leave to convention? Löser flagged a subtle danger in letting each engine pick its own ICU version. ICU does not guarantee stable orderings across releases. Two strings that compare one way in version N compare the other way in version N+1, and ordering changes have shipped in every other ICU release for years. Pinning a version at the table level protects query results but slows engine upgrades. Leaving it open speeds upgrades but risks the same query returning different results on different engines. There is no free lunch here, and the thread is working through the tradeoff in public, which is exactly what a spec discussion should look like.

Governance in the REST catalog spec kept moving through formal votes. The community opened a vote on labels for the IRC read path and a vote to formalize remote signing configuration in the REST spec. William Hyun added valuable cross-cloud research to the file-level access delegation proposal. His findings expose real operational limits in remote signing. AWS SigV4 signatures expire fifteen minutes after their timestamp, and GCS enforces the identical constraint for header signing. Azure is the harder problem. Azure Blob Storage and ADLS Gen2 treat a Shared Access Signature strictly as a token appended to the resource URI as query parameters, so true remote header signing on Azure forces a catalog into the legacy SharedKey scheme. Spec authors now have concrete evidence that one access delegation mode does not fit all three clouds, which strengthens the case for keeping multiple modes in the spec.

Several smaller threads deserve mention because they connect to the bigger stories above. A new discussion on Flink equality delete to DV conversion opened, extending the exact work Michels described in the deprecation thread. The two conversations reinforce each other. The deprecation plan only lands if the conversion path is solid, and the conversion path gains urgency from the deprecation plan. A breaking change discussion on AvroSchemaUtil flagged fixed behavior around LocalTimestamp and Timestamp mapping, a reminder that even bug fixes carry compatibility weight when a library sits under this many engines. On the reader internals side, a proposal to integrate EagerInputFile into the manifest reader targets metadata read paths, and a discussion on using Iceberg sort order metadata for read and compaction improvements in Spark asks how engines can exploit ordering information the table already declares. Sort order metadata is one of the most underused parts of the spec, and Spark putting it to work for pruning and compaction planning benefits every table that declares an order.

The REST security surface got incremental attention beyond the big votes. A review request for PR 16507 adds structured exceptions for OAuth2 token endpoint errors in the API and Core modules, small work that pays off in clearer failure modes for every REST catalog client. And in a sign of where the ecosystem is heading, a contributor introduced an OpenCrawling connector that bridges Iceberg with enterprise AI and RAG workloads. Announcements like this one used to be rare on the dev list. Now RAG pipelines treating Iceberg tables as a retrieval substrate show up alongside spec votes, which says a lot about who consumes the format in 2026.

Two community notes closed out the week. Gang Wu proposed enabling ASF-managed GitHub Copilot code review starting with iceberg-cpp as a trial, following Arrow's lead. Reviewer bandwidth is limited on the C++ repo, and a first-pass automated review buys human reviewers time for design questions. The thread drew thirteen messages, the most of any Iceberg thread this week, which shows how much the community cares about getting AI-assisted review policy right. Wu grounded the proposal in process, and that framing is worth copying. ASF Infra documents Copilot review as a supported .asf.yaml feature, Arrow already enabled and tuned it, and the .asf.yaml docs tell projects to discuss workflow and resource impact before flipping the switch. So Wu brought the discussion to the list first, scoped the trial to a single repo, and asked for objections before touching configuration. Open source projects everywhere are working out how AI review fits their norms right now, and the ones that treat it as a governance question rather than a tooling default will end up with policies their contributors trust. Expect the results of the iceberg-cpp trial to inform the main repo's decision, and expect other lakehouse projects to cite this thread when their turn comes. And the Apache Iceberg Meetup in Austin landed on July 23, giving the Texas community a chance to talk through all of the above in person.

Apache Polaris

Polaris posted the busiest week of any project on this list with 89 messages, and the center of gravity was the persistence layer. Dmitri Bourlatchkov opened a discussion on consistent multi-object changes in Polaris persistence after PRs from Ayush and Prithvi surfaced consistency issues in the JDBC backend. Bourlatchkov argued that incremental fixes work but the moment calls for a broader review. His list of requirements reads like a design charter for catalog persistence. The system needs concurrent and consistent changes where the service validates current state before committing, so renames catch name clashes. It needs independent but consistent changes to RBAC grants and metastore entities, because external authorizers like OPA and Ranger depend on that separation. It needs atomic changes across multiple entities, authorization-based filtering of list operations, credential-vending decisions rooted in exact catalog state, and server-side retries for transient failures like serialization conflicts in the database. Whatever design emerges must work across in-memory, JDBC, and NoSQL backends alike. This thread deserves attention from anyone building on Polaris, because every guarantee the catalog offers upward rests on the answers here.

The related infrastructure debates were just as pointed. In the Polaris-managed JDBC datasource discussion, Alexandre Dutra pushed back hard on a design that alternates between a Hikari connection pool and an Agroal pool depending on configuration. His objection is operational. Bugs, performance behavior, and configuration issues then vary across deployments based purely on which pool happens to be underneath. If the goal is a runtime-driven architecture, he argued, commit to it fully by removing the Quarkus datasource dependency and switching to Hikari without conditions. Half-migrations create support burdens that outlast the code. Nearby, contributors debated making the relational JDBC schema name configurable and deprecating TreeMapMetaStore and friends for removal, both signs of a codebase shedding early scaffolding as production usage grows.

API semantics got a formal decision this week. After a discussion on the right status code for table and view rename conflicts, the community moved to a vote on returning 503 Service Unavailable when a rename hits a concurrent modification of the target entity. Status code debates look small from the outside, but clients build retry logic around these codes, so getting the semantics right once beats patching client libraries forever.

Jean-Baptiste Onofré kept several strategic threads moving at once. He pushed to advance the Polaris Directories proposal, promising a revision that folds in community comments and proposing a dedicated meeting to align contributors and make it happen. He announced preparation for the Polaris 1.7.0 release, volunteering as release manager with a plan to cut the release in the last week of July and hold the project's near-monthly cadence. And he resumed the Open Sharing APIs discussion with a concrete framing of what data sharing means in Polaris terms. The goal is zero-copy sharing of live data through shared metadata and authorization. A share becomes a dedicated catalog or a catalog role scoped to specific namespaces. A recipient maps to a principal and principal role. A consumer profile is a catalog URI plus OAuth2 credentials. Credential vending hands the consumer temporary STS credentials for direct file access, and the Polaris IRC layer exposes the metadata. Polaris-to-Polaris sharing rides on federation. Onofré's point is that most of the underlying capability already exists, and what the project needs is clear packaging around the use case. That is a notable position: data sharing as product framing on top of existing catalog primitives rather than a new protocol.

The semantic layer conversation advanced too. In the Semantic Model REST API payload discussion, Bourlatchkov suggested to Yufei Gu that the REST response wrap the semantic payload in an envelope, so plain JSON and non-JSON formats both fit inside the same response structure. This matters beyond Polaris. As semantic models become catalog-managed assets, the payload representation determines which tools read and write them, and an envelope keeps the door open. The Polaris Tag Spec design proposal reached community review in parallel, and an OpenLineage follow-up kept lineage integration on the table. Add the Polaris Terraform Provider discussion and a question about vended credential passthrough in federated catalogs, and the picture is a catalog project maturing along every axis at once: persistence, API design, governance metadata, and operations tooling.

A cluster of spec-adjacent threads filled in the edges. A discussion on supporting staged creates in multi-table transactions through commitTransaction pushes Polaris toward richer atomic operations, which lines up directly with Bourlatchkov's persistence charter. A proposal on standardizing vended credential property names tackles a small but painful interoperability gap, since every engine that consumes vended credentials today handles naming quirks with adapter code. Two threads continued the Iceberg table encryption conversation, including a survey of the current state of table encryption in Polaris. Read those next to the Iceberg encrypted deletion-vector thread above and a clear picture emerges: encryption at the format layer is real enough now that catalogs must decide what they manage and what they pass through.

Observability and internals work continued in parallel. A proposal for REST endpoints exposing table metrics and events gives operators a standard way to read activity out of the catalog, and a related thread discussed making catalog ID nullable in the JDBC events tables. A refactoring discussion on removing PolarisMetricsManager from PolarisMetaStoreManager separates concerns that had grown together, and a conversation on Polaris SPI principles works toward stated rules for the project's extension points. Test infrastructure got attention too, with a decision thread on moving Spark plugin regression tests from Docker to JUnit, a change that shortens the feedback loop for every contributor touching the Spark integration.

One practical note from the contributor workflow. Bourlatchkov hit a GitHub limitation on the principal properties PR thread where a force-pushed branch blocked reopening PR 4405. His advice to Prithvi was pragmatic. Skip the fight with GitHub, open fresh PRs for the stuck changes, and cross-reference them. Ten messages on this thread show how much active review is flowing through the project right now.

Apache Arrow

Arrow delivered a release week across the board. Raúl Cumplido announced Apache Arrow 25.0.0, which closes 222 resolved issues since 24.0.0. The subprojects matched that pace. Arrow JS voted on and released 21.2.0, Arrow Go passed 18.7.0, Arrow Rust approved 58.4.0, and the Rust object store crate put 0.14.1 up for vote. Four language ecosystems cutting releases in one week is the payoff of Arrow's decentralized release model, where each implementation ships on its own cadence instead of waiting on a monolithic train.

The most interesting technical thread came from Antoine Pitrou, who raised a deceptively simple question: what does the nullable flag actually mean for non-nullable fields with non-trivial types? His examples cut deep. Consider a non-nullable child field inside a struct where the parent has nulls, which makes some child elements logically null. Or a dictionary array with nulls in the dictionary values but none in the indices. Or a union or run-end-encoded array with nulls in child values. Arrow C++ has historically treated nullable as metadata with no semantic force, and Pitrou asked what other implementations do and what they should do. Cross-implementation consistency questions like this one are where interoperability lives or dies, because two implementations that disagree on null semantics will silently produce different results from the same bytes.

It is worth pausing on what 25.0.0 represents. Arrow long ago stopped being just a memory format and became the connective layer of the analytics stack. The columnar representation, the IPC and streaming machinery, the Flight RPC layer, and the compute kernels all ship in this release across C++, Python, Java, and the rest of the bindings. Every project covered in this newsletter touches Arrow somewhere. DataFusion is built on arrow-rs. Iceberg Rust reads into Arrow batches. Parquet readers in most engines decode straight into Arrow memory. A 222-issue release here quietly upgrades the floor under all of them. The independent JS, Go, and Rust release trains matter for the same reason. A fix in the Go Parquet reader or the JS table implementation reaches users in days, not quarters, because no subproject waits on another.

David Li shared that contributors have overhauled the ADBC documentation ahead of the next release. The rewrite tackles a real source of confusion. Most ADBC drivers now come from vendors and groups outside the ASF, and users struggle to understand which drivers come from where. The new docs explain the different sources directly. This reflects a healthy pattern for a database connectivity standard: the spec lives at the ASF while the driver ecosystem grows around it, and the docs now match that reality.

Apache Parquet

Parquet had a decision-heavy week, with two format votes and a release candidate all in flight. Prateek Gaur opened the vote to add ALP encoding to the Parquet format. ALP, short for Adaptive Lossless floating-Point, compresses floating point data far better than the general-purpose encodings Parquet has today, and this proposal arrives with unusual rigor. Reference implementations exist in parquet-java, Arrow C++, and Arrow Go, plus test artifacts in parquet-testing. Cross-language compatibility is verified, with the Arrow C++ decoder reading Java-written data across V1 and V2 pages, multiple vector sizes, and several real datasets with zero mismatches. The twelve message vote thread shows strong engagement. Floating point columns dominate ML feature tables and sensor data, so a purpose-built encoding lands right where modern workloads hurt.

The release train moved alongside the format work. Fokko Driesprong proposed Parquet 1.18.0 RC1 with the full tarball, checksums, staged Nexus artifacts, and changelog, and the vote thread reached fourteen messages, the most active thread on the Parquet list this week. Meanwhile the vote to introduce a new File logical type collected support, with Kevin Liu among the +1s, and the result thread confirmed it passed. A File logical type gives engines a shared way to represent file references inside Parquet data, which matters for multimodal and document-heavy datasets where tables point at binary assets.

The timestamp precision discussion showed the spec community thinking years ahead. In the extended precision nanosecond timestamps thread, Micah Kornfield worked through the physical representation question with care. For milliseconds and microseconds, the existing integer physical types already fit, so extra engineering there buys nothing. For nanoseconds, a fixed-length byte array of eight bytes technically works. But Kornfield pointed out that systems are moving toward picoseconds, with BigQuery already returning full-precision picosecond values through its storage API, and femtoseconds have come up in conversation. A nine-byte fixed-length array covers all those ranges with the same code. His position: extend the spec once with headroom instead of once per precision level.

The vector storage story from the Iceberg section continued here. In the FIXED_SIZE_LIST logical type discussion, Gunnar Morling shared results from a fixed-length list fast path implemented in Hardwood. The technique scans the encoded definition and repetition level streams, detects lists that are effectively fixed length, and bypasses standard Dremel reconstruction. For larger lists such as 768-element vector embeddings, read times drop to the level of a flat column holding the same data. Morling published a full blog post on the approach and framed it as an interim read-side win until a native FIXED_SIZE_LIST type lands in the spec. Embedding columns are becoming a first-class citizen of the analytics stack, and both the spec track and the implementation track are responding.

Reader behavior questions surfaced twice, and both are worth a moment. One thread asked about the expected behavior of older parquet-java readers on VARIANT columns. VARIANT is one of the newest logical types in the format, and the question of what a 1.x reader from two years ago does when it meets a VARIANT column is exactly the kind of forward compatibility issue that separates a durable format from a fragile one. Another thread floated an idea for passing a known file length into HadoopInputFile. Object stores charge a round trip for every metadata call, so a reader that already knows the file length from a manifest, as Iceberg readers do, saves a HEAD request per file by passing it down. Small API, real savings at scale.

The versioning conversation continued as its own track. The community worked on scheduling ad hoc syncs for Parquet versioning, discussed the shape of an eventual versioning vote, and noted that the next footer sync was canceled in favor of consolidated scheduling. With ALP, the File type, FIXED_SIZE_LIST, and extended timestamps all in flight, the question of how readers and writers negotiate feature support is no longer academic. A thread also asked whether parquet should publish a test helpers artifact, which pairs naturally with the cross-language verification culture the ALP vote showed off.

Developer workflow threads filled out the week. Divjot Arora proposed inlining parquet.thrift into parquet-java, because the current pinned dependency on parquet-format makes it impossible to validate PRs that build on merged but unreleased format features. The arrow-rs and Arrow C++ projects already vendor the thrift file for exactly this reason, and Arora asked whether parquet-java should follow. The community also worked on scheduling ad hoc syncs for the Parquet versioning discussion, keeping the bigger question of how the format itself gets versioned on a steady track.

Apache DataFusion

DataFusion packed release and community news into a compact week. Matt Butrovich proposed the DataFusion 54.1.0 release, and the seven message vote thread carried it through to a passing result. The subprojects kept pace. Ballista 54.0.0 passed its vote for the distributed execution layer, and Comet 0.17.1 cleared its release candidate for the Spark acceleration plugin. Three coordinated releases across the query engine, its distributed runtime, and its Spark integration show a project that has industrialized its release process.

The version number tells its own story. DataFusion is at 54, Ballista aligned itself to the same major line, and Comet tracks its own cadence against Spark compatibility windows. The engine now sits under a growing roster of commercial and open source query products, and its release discipline is a big reason why. Downstream projects plan against a predictable monthly-ish core release, pick up performance work quickly, and pin when they need stability. For lakehouse practitioners, the practical takeaway is that the Rust query stack, from Arrow memory through DataFusion planning to Iceberg Rust table access, now versions and ships like mature infrastructure.

The best news of the week was human. Andrew Lamb announced on behalf of the PMC that Adam Gutglick has become a DataFusion committer, and the congratulations thread ran seven messages deep. Committer growth is the leading indicator of project health, and DataFusion keeps adding maintainers as more query engines, including several commercial products, build on top of it. If you have followed this newsletter, you know DataFusion also anchors the Iceberg Rust story, so strength here compounds across the ecosystem.

Apache Ossie

The incubating Ossie project, which defines an open semantic model interchange specification, had one of its most revealing weeks yet. For readers new to it, Ossie standardizes how tools describe semantic models: the datasets, fields, metrics, relationships, and business definitions that sit between raw tables and the people or agents querying them. Every BI tool and semantic layer vendor invented its own private representation of this information, which is why moving a metric definition between tools remains a rewrite instead of an export. Ossie aims to be the shared format that ends the rewrite, the same role Iceberg played for table metadata. This week the threads split between engineering hygiene and identity questions, and both matter for a spec this young. With 47 messages, the incubating list out-talked Arrow, which says something about the appetite for a standard here.

On the identity side, a community member asked the project to clarify Ossie's relationship with FIBO and the financial services semantic stack. The question itself maps the layers well. FIBO acts as a financial reference ontology, a formal OWL and RDF conceptual model of financial concepts. Ossie acts as a semantic model and interchange layer, a YAML and JSON oriented spec for exchanging datasets, fields, metrics, relationships, ontology concepts, mappings, and AI context across tools. BI tools, AI agents, catalogs, query engines, and governance platforms then consume or produce Ossie models, with mappings to reference ontologies like FIBO where needed. The asker wanted confirmation that Ossie complements rather than replaces FIBO, and the recently announced Financial Services Semantic Working Group makes that layering question timely. In the same spirit, Justin Talbot asked the project to state its maturity and stability somewhere public. He wants the website or repo to describe the scope of expected future changes and which use cases fit the spec today. That is exactly the question early adopters ask before betting on an incubating standard, and answering it well is how incubating projects convert curiosity into adoption.

The spec work itself advanced on two fronts. Will Pugh shared that the expression language group has drafted foundational semantics for OSI and proposed a three-part plan: gather general feedback on the spec document, start landing a reference implementation right away so the semantics get evaluated in code, and bring a markdown version of the foundational semantics into the repo. The group has already worked through semantics for level-of-detail calculations, filter exclusion, and fine-grained join specifications, but Pugh proposed leaving those out of the first push and adding them later. Shipping a small core with a reference implementation beats shipping a large spec nobody has run.

The versioning discussion for semantic models produced one of the more thoughtful contributions of the week. A contributor suggested time-based versioning where each published snapshot carries an ISO 8601 timestamp, a stable model identifier, and a content hash. The combination gives ordering, uniqueness, integrity, and portability, while the timestamp alone stays insufficient as identity since timestamps collide and historical versions get imported later. The proposal went further, noting that a time-ordered revision history lets downstream systems use age as a relevance signal for versioned artifacts like verified queries and semantic mappings, with a configurable decay function weighting older artifacts down. Semantic models feeding AI agents will need exactly this kind of freshness reasoning, and it is encouraging to see it designed into the interchange layer early. A related thread on natural language predicates explored verbalizations on relationships, including the harder case of reified many-to-many-to-many relationships that need multiple readings, like a cinema session that verbalizes as a film showing at a cinema at a time from three different angles.

Several design threads showed the community sweating the vocabulary of the spec itself, which is where interchange standards win or lose. A discussion on top level metrics versus dataset level measures worked through where aggregations belong in the model and what each word means, a distinction every BI tool draws differently today. A thread on native support for units asked whether the spec should carry units of measure as first class metadata, so a revenue field knows it is dollars and a duration field knows it is milliseconds before any tool guesses. And a pointed discussion argued the spec should not prescribe AI Context as a key name. The argument matters more than the key. Baking today's AI terminology into a durable interchange format ages badly, and neutral naming keeps the spec useful when the tooling fashions change. A Data Semantic Exploration Compiler discussion hinted at tooling that consumes the models programmatically, and the project even published a shared Google Calendar for meetings and events, a small step that makes an incubating community much easier to join.

On the hygiene side, Yong Zheng proposed standardizing the naming of Python converters, the busiest Ossie thread of the week at eleven messages. The project now has five converters, spanning dbt, Omni, Honeydew, Snowflake, and GoodData, and the names vary between apache-ossie prefixes and osi suffixes. Zheng suggested converging on an apache-ossie-xxxxx rule and introducing a base converter abstraction, since three of five converters follow a common two-file pattern and the rest improvise. Parallel threads on adding CI for components, cleaning up unused directories, missing tests for the Python models, and unified linting show a project doing the unglamorous work that turns a spec repo into a dependable codebase. New contributors kept arriving too, with introductions on the list and a question about joining the Catalog Integration working group, which is the group most relevant to readers of this newsletter since it connects semantic models to catalogs like Polaris.

Cross-Project Themes

Three threads of connective tissue stood out this week. First, the AI workload pull is now visible in the format layer itself, not just the tooling around it. Iceberg's vector type discussion leans on Parquet's fixed-size list work, Gunnar Morling's Hardwood fast path makes 768-element embeddings read like flat columns, and ALP encoding targets the floating point data that ML pipelines produce in bulk. The formats are being reshaped from below by embeddings and features, and the projects are coordinating the work rather than forking it.

Second, the ecosystem is converging on infrastructure-as-code and shared operational tooling. Iceberg voted on a Terraform provider, Polaris discussed one, and Iceberg debated ASF-managed Copilot review following Arrow's adoption. The lakehouse stack is picking up the operational conventions of mainstream platform engineering, and choices made in one project ripple to the next within weeks.

Third, this was a week of guarantee-setting. Iceberg's collation thread asked what ordering guarantees the format owes engines. Arrow's nullable thread asked what null semantics implementations owe each other. Polaris's persistence thread asked what consistency the catalog owes everything above it. Ossie's maturity thread asked what stability an incubating spec owes adopters. The stack is old enough now that the hard questions are contracts, not features.

Fourth, encryption is becoming a cross-cutting concern rather than a single project's feature. Iceberg users are hitting real bugs on encrypted V3 tables and asking for patch releases. Polaris contributors are mapping what table encryption support means for a catalog that vends credentials and exposes metadata. William Hyun's cross-cloud signing research shows that even the access delegation machinery underneath encryption behaves differently on AWS, GCS, and Azure. Teams with regulated data should read these threads together, because the encryption story spans format, catalog, and cloud provider, and gaps at any layer surface as production incidents at another.

Fifth, and quietly, Terraform showed up on two lists in the same week. The Iceberg community is voting on its provider's first release while Polaris discusses starting one. Neither thread references the other, but they answer the same demand. Platform teams manage warehouses, buckets, and IAM through infrastructure-as-code today, and catalogs plus table resources are the missing piece. Expect the two providers to converge on similar resource models, and expect users to push for that convergence once both exist.

Looking Ahead

Watch for the ALP encoding and Parquet 1.18.0 vote results, the Iceberg Terraform provider's first release, and JB's timeline for cutting Polaris 1.7.0 in the final week of July. The Iceberg equality delete deprecation thread will keep drawing V4 design energy, and the promised Polaris Directories meeting should set that proposal's direction. On the Ossie side, look for the foundational semantics feedback round and the converter naming decision.

A few slower-burning items belong on the radar too. The Jackson 3 migration discussion in Iceberg will resurface every time a downstream framework drops Jackson 2 support, so the sequencing decision gets harder with each quarter of delay. The Vortex file format criteria document is a living draft, and the criteria half of it will shape every future format proposal regardless of what happens to Vortex itself. And the Ossie Catalog Integration working group is the thread most likely to pull this newsletter's two halves together, since semantic models stored and served through catalogs like Polaris is where the lakehouse and semantic layer stories merge. The answers to this week's contract questions, on collation, nullability, and persistence consistency, will take longer than any release cycle, and they will matter more than any single release.

Resources & Further Learning

Get Started with Dremio

Free Downloads

Books by Alex Merced