Modern enterprises possess a vast amount of unstructured data, yet they frequently encounter significant challenges in managing and extracting value from it. Historically, unlocking the insights hidden within PDFs, audio files, images, and unstructured text required a fragmented architecture: moving data out of your warehouse, stitching together complex LLM pipelines, and managing disparate search indexes.

BigQuery has worked with many enterprises to make sense of their unstructured data sources. For example, consider an advanced healthcare company managing thousands of clinical trial documents in PDF form. BigQuery helps unlock insights from these documents through a simple, five-step lifecycle: Access, Process, Ground, Relate, and Activate.

In this post, we are highlighting three major milestones focused heavily on the "Ground" phase of this framework:

  1. General Availability (GA) of Autonomous Embedding Generation

  2. General Availability (GA) of AI.SEARCH with massive single-query performance gains

  3. Public Preview of Hybrid Search

Let’s dive into how these features work together to simplify your AI architecture, using a real-world clinical trial research platform as an example.

Simplify Pipelines with Autonomous Embedding Generation (GA)

Building a retrieval-augmented generation (RAG) pipeline or search application usually requires managing complex, asynchronous embedding infrastructure. You have to handle retries, error logging, and pipeline orchestration every time a new record arrives.

With the General Availability of Autonomous Embedding Generation, BigQuery manages this entirely for you. By simply defining a column in your schema, BigQuery asynchronously and continuously generates embeddings as new data is ingested. You have the flexibility to choose external models (like Vertex AI text-embeddings) or natively utilize Gemma embedding models directly within BigQuery.

How it works in practice:

Imagine you are building a research platform analyzing clinical trial PDFs stored in Google Cloud Storage. After extracting the study titles and disease areas into a table, you can automatically embed those titles:

BigQuery eliminates the need for complex third-party vector databases by managing enterprise-scale processing with one configuration. This autonomous embedding generation keeps data synchronized automatically as source text changes, removing the need for manual machine learning pipelines. This integrated approach streamlines workflows for dynamic datasets and reduces the operational burden of maintaining custom data scripts.

Finally, with this GA launch, Autonomous Embedding Generation now also supports generating embeddings natively over images using ObjectRefs, unlocking true multimodal search and analytics.

Natural Language Search at Scale with AI.SEARCH (GA)

To truly enable conversational analytics agents and snappier user experiences, your underlying search infrastructure needs to be intuitive and performant.

Once your data is seamlessly embedded, you need an efficient way to query it. Today, we are announcing the General Availability of AI.SEARCH(). This function provides a streamlined, natural-language-focused search experience, allowing you to easily find semantically related records without generating embeddings in your search path. In pairing this with the Autonomous Embedding Generation, we leverage the same embedding model used in your dataset for easier use. 

Furthermore, as part of efficiency investments in the last year, we have heavily optimized AI.SEARCH for single-query execution. For online applications and single-query searches (those most common in agentic searches), we have observed up to a 133x gain in slot efficiency..

This means you can serve highly concurrent, user-facing natural language searches directly out of BigQuery faster and more cost-effectively than ever before.

Unifying Keyword and Vector with Hybrid Search (Public Preview)

Semantic (vector) search is incredibly powerful; for example, the query above will successfully return conceptually related terms like "chemotherapy." However, semantic search isn't always enough. What if a researcher is searching for a specific, highly technical immunotherapy drug designation like "MK3475"? Because this alphanumeric string lacks broad semantic meaning, pure vector search might struggle to rank it correctly.

By merging lexical search with existing semantic capabilities, BigQuery's hybrid search allows for data retrieval based on both keyword similarity and underlying meaning. This approach unites the conceptual depth of semantic vector search with the pinpoint accuracy of lexical matching, utilizing algorithms such as Reciprocal Rank Fusion and BM25. The result is a significant boost in search precision and a reduction in LLM hallucination costs through the reranking of results based on keyword frequency and semantic relevance. Users can implement this via the AI.SEARCH and VECTOR_SEARCH functions by employing the hybrid mode or lexical_search_columns parameters. Furthermore, performance can be optimized by extending vector indexes to include keyword data, which accelerates the lexical search process.

You can now perform hybrid searches effortlessly using the AI.SEARCH() function by simply setting the mode to HYBRID:

To speed up these hybrid queries at scale, you can easily extend your CREATE VECTOR INDEX DDL to include the keyword columns you want to use for the lexical portion of the search, natively combining your indexes.

Building an End-to-End Unstructured Data Analytics Platform

The search and embedding features launching today are part of a much broader vision. We are building an end-to-end unstructured data analytics platform. One common type of unstructured data is documents, and BigQuery now provides the complete toolset to manage this workflow from end to end:

https://storage.googleapis.com/gweb-cloudblog-publish/original_images/image1_bQnyG2Q.gif

  • Access: Enable zero-ETL workflows by querying unstructured PDFs and documents directly where they live in Google Cloud Storage using Object Tables.

  • Process: Utilize embedded AI capabilities like AI.PARSE_DOCUMENT (coming soon) for layout-aware chunking (perfect for RAG), AI.GENERATE for entity extraction, and AI.CLASSIFY to instantly categorize records using foundational models directly in your SQL pipelines.

  • Ground: Build highly accurate context using Autonomous Embeddings and Hybrid Search. Hybrid search combines the conceptual understanding of semantic vector search with the exact precision of lexical keyword matching By reranking results based on both semantic relevance and keyword frequency, you drastically increase search precision and drive down LLM hallucinations.

  • Relate: Uncover hidden multi-hop insights by mapping extracted entities (like Sponsors, Trials, and Drugs) into a BigQuery Graph—no specialized graph database required.

  • Activate: Bring it all together with BigQuery's Conversational Analytics agents. Using functions like AI.AGG, you can chat directly with your complex data, generate visualizations, and perform trend analysis at massive scale.

With unstructured data as a first-class citizen in BigQuery, you can finally bridge the gap between your raw documents and conversational AI.

Ready to get started?

Posted in