Google BigQuery Vector Search

Google Cloud BigQuery Vector Search lets you use GoogleSQL to do semantic search, using vector indexes for fast approximate results, or using brute force for exact results.

This tutorial illustrates how to work with an end-to-end data and embedding management system in LangChain, and provides a scalable semantic search in BigQuery using theBigQueryVectorStore class. This class is part of a set of 2 classes capable of providing a unified data storage and flexible vector search in Google Cloud:

• BigQuery Vector Search: with BigQueryVectorStore class, which is ideal for rapid prototyping with no infrastructure setup and batch retrieval.
• Feature Store Online Store: with VertexFSVectorStore class, enables low-latency retrieval with manual or scheduled data sync. Perfect for production-ready user-facing GenAI applications.

Getting started

Install the library

%pip install -qU  langchain langchain-google-vertexai "langchain-google-community[featurestore]"

To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.

import IPython

app = IPython.Application.instance()
app.kernel.do_shutdown(True)

Before you begin

Set your project ID

If you don’t know your project ID, try the following:

• Run gcloud config list.
• Run gcloud projects list.
• See the support page: Locate the project ID.

PROJECT_ID = ""  # @param {type:"string"}

# Set the project id
! gcloud config set project {PROJECT_ID}

Set the region

You can also change the REGION variable used by BigQuery. Learn more about BigQuery regions.

REGION = "us-central1"  # @param {type: "string"}

Set the dataset and table names

They will be your BigQuery Vector Store.

DATASET = "my_langchain_dataset"  # @param {type: "string"}
TABLE = "doc_and_vectors"  # @param {type: "string"}

Authenticating your notebook environment

• If you are using Colab to run this notebook, uncomment the cell below and continue.
• If you are using Vertex AI Workbench, check out the setup instructions here.

# from google.colab import auth as google_auth

# google_auth.authenticate_user()

Demo: BigQueryVectorStore

Create an embedding class instance

You may need to enable Vertex AI API in your project by running gcloud services enable aiplatform.googleapis.com --project {PROJECT_ID} (replace {PROJECT_ID} with the name of your project).

You can use any LangChain embeddings model.

from langchain_google_vertexai import VertexAIEmbeddings

embedding = VertexAIEmbeddings(
    model_name="textembedding-gecko@latest", project=PROJECT_ID
)

Initialize BigQueryVectorStore

BigQuery Dataset and Table will be automatically created if they do not exist. See class definition here for all optional paremeters.

from langchain_google_community import BigQueryVectorStore

store = BigQueryVectorStore(
    project_id=PROJECT_ID,
    dataset_name=DATASET,
    table_name=TABLE,
    location=REGION,
    embedding=embedding,
)

Add texts

all_texts = ["Apples and oranges", "Cars and airplanes", "Pineapple", "Train", "Banana"]
metadatas = [{"len": len(t)} for t in all_texts]

store.add_texts(all_texts, metadatas=metadatas)

Search for documents

query = "I'd like a fruit."
docs = store.similarity_search(query)
print(docs)

Search for documents by vector

query_vector = embedding.embed_query(query)
docs = store.similarity_search_by_vector(query_vector, k=2)
print(docs)

Searching Documents with Metadata Filters

The vectorstore supports two methods for applying filters to metadata fields when performing document searches:

• Dictionary-based Filters
  • You can pass a dictionary (dict) where the keys represent metadata fields and the values specify the filter condition. This method applies an equality filter between the key and the corresponding value. When multiple key-value pairs are provided, they are combined using a logical AND operation.
• SQL-based Filters
  • Alternatively, you can provide a string representing an SQL WHERE clause to define more complex filtering conditions. This allows for greater flexibility, supporting SQL expressions such as comparison operators and logical operators. Learn more about BigQuery operators.

# Dictionary-based Filters
# This should only return "Banana" document.
docs = store.similarity_search_by_vector(query_vector, filter={"len": 6})
print(docs)

# SQL-based Filters
# This should return "Banana", "Apples and oranges" and "Cars and airplanes" documents.
docs = store.similarity_search_by_vector(query_vector, filter="len = 6 AND len > 17")
print(docs)

Batch search

BigQueryVectorStore offers a batch_search method for scalable Vector similarity search.

results = store.batch_search(
    embeddings=None,  # can pass embeddings or
    queries=["search_query", "search_query"],  # can pass queries
)

Add text with embeddings

You can also bring your own embeddings with the add_texts_with_embeddings method. This is particularly useful for multimodal data which might require custom preprocessing before the embedding generation.

items = ["some text"]
embs = embedding.embed(items)

ids = store.add_texts_with_embeddings(
    texts=["some text"], embs=embs, metadatas=[{"len": 1}]
)

Low-latency serving with Feature Store

You can simply use the method .to_vertex_fs_vector_store() to get a VertexFSVectorStore object, which offers low latency for online use cases. All mandatory parameters will be automatically transferred from the existing BigQueryVectorStore class. See the class definition for all the other parameters you can use.

Moving back to BigQueryVectorStore is equivalently easy with the .to_bq_vector_store() method.

store.to_vertex_fs_vector_store()  # pass optional VertexFSVectorStore parameters as arguments