{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Interactive Loan Application Assistant (Financial Services)\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fmultimodal-live-api%2Freal_time_rag_bank_loans_gemini_2_0.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<br/>\n",
        "\n",
        "<br/>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "<br/>\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/real_time_rag_bank_loans_gemini_2_0.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84f0f73a0f76"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "| Author(s) | [Koushik Ghosh](https://github.com/Koushik25feb) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>\n",
        "⚠️ Gemini 2.0 Flash (Model ID: <code>gemini-2.0-flash-live-preview-04-09</code>) and the Google Gen AI SDK are currently experimental and output can vary ⚠️</b>\n",
        "</div>\n",
        "\n",
        "## Overview\n",
        "\n",
        "Navigating complex loan documents can be overwhelming for users, with lengthy terms, intricate conditions, and financial jargon that often create confusion. Our Interactive Loan Application Assistant demo app is designed to simplify this process using Gemini 2.0.\n",
        "\n",
        "By leveraging cutting-edge **text-to-text** capabilities, this solution enhances accessibility, enabling users to:\n",
        "\n",
        "- Make informed financial decisions with clarity.\n",
        "- Save time by avoiding the need for manual document review.\n",
        "- Ensure transparency and confidence when dealing with critical loan information.\n",
        "- Whether it's for individuals seeking to understand their loan agreements or financial advisors assisting clients, this demo app bridges the gap between complex documentation and user comprehension, making financial services more transparent and user-centric.\n",
        "\n",
        "This notebook provides a comprehensive demonstration of how Gemini 2.0 can act as your personal file assistant across various storage platforms (local storage, Google Cloud Storage, and Google Drive). It empowers users to seamlessly understand and interact with their loan documents. This notebook focuses on two key features:\n",
        "\n",
        "- **Retrieval Augmented Generation (RAG):**  Text  output generation grounded in provided documents.\n",
        "  - **Multimodal Live API:** Text output generation.\n",
        "\n",
        "- **Large Context Window (Entire document):**  Text  output generation for large context in one go.\n",
        "  - **Multimodal Live API:** Text output generation.\n",
        "\n",
        "\n",
        "## High-Level Flow\n",
        "\n",
        "This notebook simplifies understanding complex loan documents using Gemini. Here's how it works:\n",
        "\n",
        "1.   Initialization:\n",
        "      * Installs required libraries (like tools for reading PDFs and using Gemini).\n",
        "      * Connects to your Google Cloud account for secure access.\n",
        "      * Chooses the Gemini model for generating answers.\n",
        "\n",
        "2.   Document Processing:\n",
        "      * Loads your loan documents (from cloud storage or your computer).\n",
        "      * Divides documents into smaller, manageable chunks.\n",
        "      * Creates searchable representations of chunks for easy retrieval.\n",
        "\n",
        "3. Question Answering:\n",
        "      * You ask a question related to the loan documents.\n",
        "      * The system quickly finds relevant parts of the documents.\n",
        "      * The Gemini model combines your question and relevant information to generate an answer.\n",
        "      * The answer is presented to you in text format.\n",
        "\n",
        "4. Text Output:\n",
        "      * The answer on text output\n",
        "      * This is useful for accessibility or personal preference.\n",
        "\n",
        "5. Large Context Window:\n",
        "      * For longer documents, the entire content can be analyzed at once.\n",
        "      * This might be slower, but can provide more comprehensive answers.\n",
        "\n",
        "**More in depth technical details in the code below**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Google Gen AI SDK\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "# Install the SDK\n",
        "%pip install google-genai==0.1.0 PyPDF2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mqkS0MFt53rB"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "\n",
        "from google.colab import auth\n",
        "\n",
        "auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdvJRUWRNGHE"
      },
      "source": [
        "## Essential imports and the model configuration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5303c05f7aa6"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6fc324893334"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import subprocess\n",
        "from typing import Any\n",
        "\n",
        "from IPython.display import Audio, Markdown, display\n",
        "import PyPDF2\n",
        "import gcsfs\n",
        "from google import genai\n",
        "from google.cloud import storage\n",
        "from google.genai.types import (\n",
        "    EmbedContentConfig,\n",
        "    GenerateContentConfig,\n",
        "    LiveConnectConfig,\n",
        "    Retrieval,\n",
        "    Tool,\n",
        "    VertexAISearch,\n",
        "    VertexRagStore,\n",
        ")\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "from tenacity import retry, stop_after_attempt, wait_random_exponential"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e43229f3ad4f"
      },
      "source": [
        "### Define Gemini 2.0 model and text embedding"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cf93d5f0ce00"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.0-flash-live-preview-04-09\"  # @param {type: \"string\"}\n",
        "\n",
        "MODEL = (\n",
        "    f\"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/{MODEL_ID}\"\n",
        ")\n",
        "text_embedding_model = \"text-embedding-005\"  # @param {type:\"string\", isTemplate: true}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ibJJK_4ZfWhf"
      },
      "source": [
        "### Initialize GenAi Client\n",
        "\n",
        "* Client for calling the Vertex AI Gen AI APIs.\n",
        "* `vertexai=True`, indicates the client should communicate with the Vertex AI API endpoints."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t7Jq0XZUnDz7"
      },
      "outputs": [],
      "source": [
        "client = genai.Client(\n",
        "    vertexai=True,\n",
        "    project=PROJECT_ID,\n",
        "    location=LOCATION,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IOsTjQqBER70"
      },
      "source": [
        "## Multimodal Live API Implementation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wqCyzisyF6nK"
      },
      "source": [
        "### Authentication and token setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eBeVELQzF5-O"
      },
      "outputs": [],
      "source": [
        "def get_access_token():\n",
        "    \"\"\"Fetches the Google Cloud access token.\"\"\"\n",
        "    try:\n",
        "        return subprocess.check_output(\n",
        "            [\"gcloud\", \"auth\", \"print-access-token\"], text=True\n",
        "        ).strip()\n",
        "    except subprocess.CalledProcessError as e:\n",
        "        print(f\"Error getting access token: {e}\")\n",
        "        return None\n",
        "\n",
        "\n",
        "def get_api_endpoint():\n",
        "    \"\"\"Retrieves the API endpoint from the environment variable.\"\"\"\n",
        "    api_endpoint = os.environ.get(\"API_ENDPOINT\")\n",
        "    if not api_endpoint:\n",
        "        print(\"Error: API_ENDPOINT environment variable not set.\")\n",
        "        return None\n",
        "    return api_endpoint"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BuL_GUXiIGSc"
      },
      "source": [
        "### Multimodal Live API - Text in text out implementation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mco9ix1iIJhe"
      },
      "outputs": [],
      "source": [
        "def generate_text(prompt: str) -> str:\n",
        "    \"\"\"Generates text using the specified model and prompt.\n",
        "\n",
        "    This function utilizes the `vertex_client` to interact with the\n",
        "    Generative AI models. It takes a prompt as input and returns the\n",
        "    generated text response.\n",
        "\n",
        "    Args:\n",
        "        prompt: The input prompt for text generation.\n",
        "\n",
        "    Returns:\n",
        "        The generated text response.\n",
        "    \"\"\"\n",
        "    modality = \"TEXT\"\n",
        "    response = client.models.generate_content(\n",
        "        model=MODEL,\n",
        "        contents=f\"{prompt}\",\n",
        "        config=GenerateContentConfig(\n",
        "            response_modalities=[modality],\n",
        "        ),\n",
        "    )\n",
        "    return response.text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A0xNxpkevK8V"
      },
      "source": [
        "### Multimodel Live API - Text to Audio implementation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ARidhpu0vIre"
      },
      "outputs": [],
      "source": [
        "async def generate_n_play_audio(client, prompt):\n",
        "    \"\"\"Generates audio from text using Gemini and plays it.\n",
        "\n",
        "    Args:\n",
        "      client: The Gen AI client instance.\n",
        "      prompt: The text to convert to audio.\n",
        "      model_id: The ID of the Gemini model to use (default: 'gemini-2.0-flash-live-preview-04-09').\n",
        "\n",
        "    Returns:\n",
        "      None. Plays the generated audio directly.\n",
        "    \"\"\"\n",
        "    config = LiveConnectConfig(response_modalities=[\"AUDIO\"])\n",
        "    async with client.aio.live.connect(\n",
        "        model=MODEL_ID,\n",
        "        config=config,\n",
        "    ) as session:\n",
        "        text_input = prompt\n",
        "        display(Markdown(f\"**Input:** {text_input}\"))\n",
        "\n",
        "        await session.send(input=text_input, end_of_turn=True)\n",
        "\n",
        "        audio_data = []\n",
        "        async for message in session.receive():\n",
        "            if message.server_content.model_turn:\n",
        "                for part in message.server_content.model_turn.parts:\n",
        "                    if part.inline_data:\n",
        "                        audio_data.append(\n",
        "                            np.frombuffer(part.inline_data.data, dtype=np.int16)\n",
        "                        )\n",
        "\n",
        "        if audio_data:\n",
        "            display(Audio(np.concatenate(audio_data), rate=24000, autoplay=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7thHmNHkkJ3x"
      },
      "source": [
        "## Quick Usages\n",
        "\n",
        "Verify the initilisation with simple question"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c3T0HI02Nf3G"
      },
      "outputs": [],
      "source": [
        "test_prompt = \"How many days are there in year 2025?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFPYjSi-Fu_C"
      },
      "source": [
        "### Text In Text Out\n",
        "\n",
        "Quick verification of all setup before further proceeding"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aLXgIDu5FS7-"
      },
      "outputs": [],
      "source": [
        "output = generate_text(test_prompt)\n",
        "print(output)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_z5XFF55valp"
      },
      "source": [
        "### Text in Audio Out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RuMEPbrjvkvQ"
      },
      "outputs": [],
      "source": [
        "await generate_n_play_audio(client, test_prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0sO_veps6Xt0"
      },
      "source": [
        "## Option 1: Custom RAG (Retrieval-Augmented Generation)\n",
        "\n",
        "This section demonstrates how to build a custom RAG system using Gemini. RAG combines information retrieval (finding relevant information) with text generation (creating answers).\n",
        "\n",
        "## What is RAG?\n",
        "\n",
        "Imagine having a vast library of loan documents. RAG helps you quickly find the answers to your questions within these documents. It's like having a smart assistant that can pinpoint the specific sections you need, instead of having to read everything yourself.\n",
        "\n",
        "## **How RAG Works:**\n",
        "\n",
        "1. Get Your Documents:\n",
        "\n",
        "  * Specify the location of your loan documents (Google Cloud Storage or local files).\n",
        "\n",
        "2. Create a Searchable Index:\n",
        "\n",
        "  * The system breaks down the documents into smaller chunks of text.\n",
        "  * It then creates \"embeddings\" – numerical representations of these chunks – which allow for fast and efficient searching. Think of it like creating a detailed index for your library, making it easy to find what you're looking for.\n",
        "\n",
        "3. Ask Questions and Get Answers:\n",
        "\n",
        "  * When you ask a question, the system uses the embeddings to quickly identify the most relevant chunks of text.\n",
        "  * These chunks, along with your question, are given to the Gemini AI model, which generates an answer. The answer is based on the specific information within the documents, ensuring accuracy and relevance.\n",
        "\n",
        "\n",
        "## **Benefits of RAG:**\n",
        "\n",
        "  * Efficiency: RAG allows for faster and more targeted information retrieval compared to manual searching.\n",
        "  * Accuracy: Answers are grounded in the provided documents, reducing the risk of hallucinations or incorrect information.\n",
        "  * Flexibility: You can easily update or add new documents to the system, enhancing its capabilities over time.\n",
        "\n",
        "\n",
        "## **Example:**\n",
        "Let's say you want to know about the types of home loans available. Using RAG, you can simply ask the question, and the system will:\n",
        "\n",
        "  1. Identify the relevant sections in your loan documents.\n",
        "  2. Use Gemini to extract and generate an answer based on those sections.\n",
        "  3. Present the answer to you in a clear and concise format.\n",
        "\n",
        "This approach significantly streamlines the process of extracting information from complex documents, enabling more efficient and informed decision-making.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cx1ALgNtnMUn"
      },
      "source": [
        "#### Get your documents\n",
        "\n",
        "\n",
        "\n",
        "1.   Local Content\n",
        "2.   GCS Files\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aJclTv_l3llP"
      },
      "outputs": [],
      "source": [
        "# GCS path for the demo document, please use this for only reference.\n",
        "\n",
        "# gs://github-repo/generative-ai/gemini2/use-cases/loan_example_documents/DEMO-BANK-LOAN-DETAILS.pdf\n",
        "# gs://github-repo/generative-ai/gemini2/use-cases/loan_example_documents/Demo-bank-home-loan-agreement.pdf\n",
        "\n",
        "document = [\n",
        "    \"gs://github-repo/generative-ai/gemini2/use-cases/loan_example_documents/DEMO-BANK-LOAN-DETAILS.pdf\",\n",
        "    \"gs://github-repo/generative-ai/gemini2/use-cases/loan_example_documents/Demo-bank-home-loan-agreement.pdf\",\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tlB48y1IXj8e"
      },
      "outputs": [],
      "source": [
        "## Use and modify the below example code if you have the local document\n",
        "\n",
        "# document = [\n",
        "#     \"/content/DEMO-BANK-LOAN-DETAILS.pdf\",\n",
        "#     \"/content/Demo-bank-home-loan-agreement.pdf\"\n",
        "# ]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aEbT4aCXSFF9"
      },
      "outputs": [],
      "source": [
        "# Document read from GCS\n",
        "\n",
        "\n",
        "def extract_text_from_gcs(gcs_path):\n",
        "    \"\"\"Extracts text from a PDF file.\"\"\"\n",
        "    bucket_name = gcs_path.split(\"/\")[2]\n",
        "    file_name = \"/\".join(gcs_path.split(\"/\")[3:])\n",
        "\n",
        "    storage_client = storage.Client()\n",
        "    bucket = storage_client.bucket(bucket_name)\n",
        "    blob = bucket.blob(file_name)\n",
        "\n",
        "    document_content = blob.download_as_bytes()\n",
        "    return document_content"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mbidBvbOZRGo"
      },
      "outputs": [],
      "source": [
        "def extract_text_from_pdf(pdf_path):\n",
        "    \"\"\"Extracts text from a PDF file, skipping blanks and handling empty PDFs.\"\"\"\n",
        "    try:\n",
        "        with open(pdf_path, \"rb\") as pdf_file:\n",
        "            pdf_reader = PyPDF2.PdfReader(pdf_file)\n",
        "            text = []\n",
        "\n",
        "            # Check if PDF has any pages\n",
        "            if not pdf_reader.pages:\n",
        "                return \"Error: PDF file is empty.\"\n",
        "\n",
        "            for page_number in range(len(pdf_reader.pages)):\n",
        "                page_text = pdf_reader.pages[page_number].extract_text()\n",
        "\n",
        "                # Only append non-empty, non-blank pages\n",
        "                if page_text and not page_text.isspace():\n",
        "                    clean_text = page_text.strip()\n",
        "                    if clean_text:\n",
        "                        text.append(clean_text)\n",
        "\n",
        "            # Join the extracted text from all pages into a single string\n",
        "            final_text = \"\\n\".join(text)\n",
        "            return (\n",
        "                final_text\n",
        "                if final_text\n",
        "                else \"Error: No readable text found in the PDF.\"\n",
        "            )\n",
        "\n",
        "    except FileNotFoundError:\n",
        "        return \"Error: PDF file not found.\"\n",
        "    except PyPDF2.errors.PdfReadError:\n",
        "        return \"Error: Could not read the PDF file. It may be corrupted or encrypted.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BPoVUUQ2WBbI"
      },
      "source": [
        "#### RAG Creation\n",
        "\n",
        "RAG based on the large files chunking and embedding using text-embedding-005 with vector db"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "87dsljf4WFAH"
      },
      "outputs": [],
      "source": [
        "@retry(wait=wait_random_exponential(multiplier=1, max=120), stop=stop_after_attempt(4))\n",
        "def get_embeddings(\n",
        "    embedding_client: Any, embedding_model: str, text: str, output_dim: int = 768\n",
        ") -> list[float]:\n",
        "    \"\"\"\n",
        "    Generate embeddings for text with retry logic for API quota management.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        response = embedding_client.models.embed_content(\n",
        "            model=embedding_model,\n",
        "            contents=[text],\n",
        "            config=EmbedContentConfig(output_dimensionality=output_dim),\n",
        "        )\n",
        "        return [response.embeddings[0].values]\n",
        "    except Exception as e:\n",
        "        if \"RESOURCE_EXHAUSTED\" in str(e):\n",
        "            return None\n",
        "        print(f\"Error generating embeddings: {str(e)}\")\n",
        "        raise"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FeKzRYQSWF7a"
      },
      "outputs": [],
      "source": [
        "def build_index(\n",
        "    document_paths: list[str],\n",
        "    embedding_client: Any,\n",
        "    embedding_model: str,\n",
        "    chunk_size: int = 500,\n",
        ") -> pd.DataFrame:\n",
        "    \"\"\"\n",
        "    Build searchable index from documents with page-wise processing.\n",
        "    \"\"\"\n",
        "    all_chunks = []\n",
        "    gcs_file_system = gcsfs.GCSFileSystem(project=PROJECT_ID)\n",
        "\n",
        "    for doc_path in document_paths:\n",
        "        try:\n",
        "            with gcs_file_system.open(doc_path, \"rb\") as file:\n",
        "                pdf_reader = PyPDF2.PdfReader(file)\n",
        "\n",
        "                for page_num in range(len(pdf_reader.pages)):\n",
        "                    page = pdf_reader.pages[page_num]\n",
        "                    page_text = page.extract_text()\n",
        "\n",
        "                    chunks = [\n",
        "                        page_text[i : i + chunk_size]\n",
        "                        for i in range(0, len(page_text), chunk_size)\n",
        "                    ]\n",
        "\n",
        "                    for chunk_num, chunk_text in enumerate(chunks):\n",
        "                        embeddings = get_embeddings(\n",
        "                            embedding_client, embedding_model, chunk_text\n",
        "                        )\n",
        "\n",
        "                        if embeddings is None:\n",
        "                            print(\n",
        "                                f\"Warning: Could not generate embeddings for chunk {chunk_num} on page {page_num + 1}\"\n",
        "                            )\n",
        "                            continue\n",
        "\n",
        "                        chunk_info = {\n",
        "                            \"document_name\": doc_path,\n",
        "                            \"page_number\": page_num + 1,\n",
        "                            \"page_text\": page_text,\n",
        "                            \"chunk_number\": chunk_num,\n",
        "                            \"chunk_text\": chunk_text,\n",
        "                            \"embeddings\": embeddings,\n",
        "                        }\n",
        "                        all_chunks.append(chunk_info)\n",
        "\n",
        "        except Exception as e:\n",
        "            print(f\"Error processing document {doc_path}: {str(e)}\")\n",
        "            continue\n",
        "\n",
        "    if not all_chunks:\n",
        "        raise ValueError(\"No chunks were created from the documents\")\n",
        "\n",
        "    return pd.DataFrame(all_chunks)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "817nRd1kWQ8q"
      },
      "outputs": [],
      "source": [
        "def get_relevant_chunks(\n",
        "    query: str,\n",
        "    vector_db: pd.DataFrame,\n",
        "    embedding_client: Any,\n",
        "    embedding_model: str,\n",
        "    top_k: int = 3,\n",
        ") -> str:\n",
        "    \"\"\"\n",
        "    Retrieve most relevant document chunks for a query using similarity search.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        query_embedding = get_embeddings(embedding_client, embedding_model, query)\n",
        "\n",
        "        if query_embedding is None:\n",
        "            return \"Could not process query due to quota issues\"\n",
        "\n",
        "        similarities = [\n",
        "            cosine_similarity(query_embedding, chunk_emb)[0][0]\n",
        "            for chunk_emb in vector_db[\"embeddings\"]\n",
        "        ]\n",
        "\n",
        "        top_indices = np.argsort(similarities)[-top_k:]\n",
        "        relevant_chunks = vector_db.iloc[top_indices]\n",
        "\n",
        "        context = []\n",
        "        for _, row in relevant_chunks.iterrows():\n",
        "            context.append(\n",
        "                {\n",
        "                    \"document_name\": row[\"document_name\"],\n",
        "                    \"page_number\": row[\"page_number\"],\n",
        "                    \"chunk_number\": row[\"chunk_number\"],\n",
        "                    \"chunk_text\": row[\"chunk_text\"],\n",
        "                }\n",
        "            )\n",
        "\n",
        "        return \"\\n\\n\".join(\n",
        "            [\n",
        "                f\"[Page {chunk['page_number']}, Chunk {chunk['chunk_number']}]: {chunk['chunk_text']}\"\n",
        "                for chunk in context\n",
        "            ]\n",
        "        )\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error getting relevant chunks: {str(e)}\")\n",
        "        return \"Error retrieving relevant chunks\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6MWtTrJ-WXs7"
      },
      "outputs": [],
      "source": [
        "@retry(wait=wait_random_exponential(multiplier=1, max=120), stop=stop_after_attempt(4))\n",
        "def generate_answer(query: str, context: str, llm_client: Any, llm_model: str) -> str:\n",
        "    \"\"\"\n",
        "    Generate answer using LLM with retry logic for API quota management.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        # If context indicates earlier quota issues, return early\n",
        "        if context in [\n",
        "            \"Could not process query due to quota issues\",\n",
        "            \"Error retrieving relevant chunks\",\n",
        "        ]:\n",
        "            return \"Can't Process, Quota Issues\"\n",
        "\n",
        "        prompt = f\"\"\"Based on the following context, please answer the question.\n",
        "        Include page number and chunk number in your citations when referring to specific information.\n",
        "\n",
        "        Context:\n",
        "        {context}\n",
        "\n",
        "        Question: {query}\n",
        "\n",
        "        Answer:\"\"\"\n",
        "\n",
        "        response = llm_client.models.generate_content(model=llm_model, contents=prompt)\n",
        "        return response.text\n",
        "\n",
        "    except Exception as e:\n",
        "        if \"RESOURCE_EXHAUSTED\" in str(e):\n",
        "            return \"Can't Process, Quota Issues\"\n",
        "        print(f\"Error generating answer: {str(e)}\")\n",
        "        return \"Error generating answer\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zXDUqw9IWYey"
      },
      "outputs": [],
      "source": [
        "def rag(\n",
        "    document_name: str,\n",
        "    question_set: list[dict],\n",
        "    vector_db: pd.DataFrame,\n",
        "    embedding_client: Any,\n",
        "    embedding_model: str,\n",
        "    llm_client: Any,\n",
        "    top_k: int,\n",
        "    llm_model: str,\n",
        ") -> pd.DataFrame:\n",
        "    \"\"\"\n",
        "    RAG Pipeline.\n",
        "\n",
        "    Args:\n",
        "        document_name: Name of the document being queried\n",
        "        question_set: List of question dictionaries\n",
        "        vector_db: DataFrame containing document chunks and embeddings\n",
        "        embedding_client: Client for accessing embedding API\n",
        "        embedding_model: Name of the embedding model\n",
        "        llm_client: Client for accessing LLM API\n",
        "        top_k: Number of relevant chunks to retrieve (default: 3)\n",
        "\n",
        "    Returns:\n",
        "        DataFrame containing questions and generated answers\n",
        "    \"\"\"\n",
        "    results = []\n",
        "\n",
        "    for question in question_set[:1]:\n",
        "        try:\n",
        "            # Get relevant context for question\n",
        "            relevant_context = get_relevant_chunks(\n",
        "                question[\"question\"],\n",
        "                vector_db,\n",
        "                embedding_client,\n",
        "                embedding_model,\n",
        "                top_k=top_k,\n",
        "            )\n",
        "\n",
        "            # Generate answer using LLM\n",
        "            generated_answer = generate_answer(\n",
        "                question[\"question\"], relevant_context, llm_client, llm_model\n",
        "            )\n",
        "\n",
        "            # Store results\n",
        "            results.append(\n",
        "                {\n",
        "                    \"document_name\": document_name,\n",
        "                    \"question\": question[\"question\"],\n",
        "                    \"source_page_num\": question[\"page\"],\n",
        "                    \"answer\": question[\"answer\"],\n",
        "                    \"generated_answer\": generated_answer,\n",
        "                }\n",
        "            )\n",
        "\n",
        "        except Exception as e:\n",
        "            print(f\"Error processing question '{question['question']}': {str(e)}\")\n",
        "            results.append(\n",
        "                {\n",
        "                    \"document_name\": document_name,\n",
        "                    \"question\": question[\"question\"],\n",
        "                    \"source_page_num\": question[\"page\"],\n",
        "                    \"answer\": question[\"answer\"],\n",
        "                    \"generated_answer\": \"Error processing question\",\n",
        "                }\n",
        "            )\n",
        "\n",
        "    return pd.DataFrame(results)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iCn8euT3Wh0r"
      },
      "outputs": [],
      "source": [
        "vector_db_mini_vertex = build_index(\n",
        "    document, embedding_client=client, embedding_model=text_embedding_model\n",
        ")\n",
        "vector_db_mini_vertex.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5yiwVFC_Z0Vj"
      },
      "outputs": [],
      "source": [
        "question_set_1 = [\n",
        "    {\n",
        "        \"question\": \"What are the loan products available?\",\n",
        "        \"answer\": \"Home Loan, Smart Loan, Loan Against Property, Smart loan against property\",\n",
        "        \"page\": 6,\n",
        "    },\n",
        "    {\n",
        "        \"question\": \"How much is the Processing fee for the loan?\",\n",
        "        \"answer\": \"1% of the sanctioned loan amount or 10000 INR, which ever is higher\",\n",
        "        \"page\": 7,\n",
        "    },\n",
        "    {\n",
        "        \"question\": \"Documents to submit as proof od identity?\",\n",
        "        \"answer\": \"Passport, Election/voters IDs, Permanent Driving license, permanent account number, Adhaar card\",\n",
        "        \"page\": 2,\n",
        "    },\n",
        "    {\n",
        "        \"question\": \"How many days it take for Loan Pay Order?\",\n",
        "        \"answer\": \"1 day\",\n",
        "        \"page\": 5,\n",
        "    },\n",
        "    {\n",
        "        \"question\": \"Phone number for phone banking service?\",\n",
        "        \"answer\": \"+91-49-3111-1111\",\n",
        "        \"page\": 16,\n",
        "    },\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gc3TwXS8Wks6"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "\n",
        "results_df_vertex = rag(\n",
        "    document_name=document[0].split(\"/\")[-1],\n",
        "    question_set=question_set_1,\n",
        "    vector_db=vector_db_mini_vertex,\n",
        "    embedding_client=client,  # For embedding generation\n",
        "    embedding_model=text_embedding_model,  # For embedding model\n",
        "    llm_client=client,  # For answer generation,\n",
        "    top_k=10,\n",
        "    llm_model=MODEL,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NsLTh_J2RznY"
      },
      "source": [
        "### Loan QnA with Gemini 2.0 Model - RAG"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tfB3BQIrSDXv"
      },
      "outputs": [],
      "source": [
        "question = \"What are different types of home loan?\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mOFE7VFTa9R3"
      },
      "outputs": [],
      "source": [
        "relevant_context = get_relevant_chunks(\n",
        "    question, vector_db_mini_vertex, client, text_embedding_model, top_k=10\n",
        ")\n",
        "rag_prompt = f\"\"\"Based on the following context, please answer the question.\n",
        "\n",
        "Context:\n",
        "{relevant_context}\n",
        "\n",
        "Question: {question}\n",
        "\n",
        "Answer:\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vcmD3qWJniMv"
      },
      "source": [
        "#### Text output - RAG"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dQ2e3ZXEm1D7"
      },
      "outputs": [],
      "source": [
        "response = generate_text(prompt=rag_prompt)\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1NVC8L0nLmRB"
      },
      "source": [
        "#### Audio Output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1hQ7JBwtLlMj"
      },
      "outputs": [],
      "source": [
        "await generate_n_play_audio(client, rag_prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5KLDaAOWGge"
      },
      "source": [
        "## Option 2 : Large Context Window\n",
        "\n",
        "This section demonstrates Gemini's ability to handle larger documents for more comprehensive question answering.\n",
        "\n",
        "## **What is a Large Context Window?**\n",
        "\n",
        "Think of a context window as the amount of information a model can consider at once. With a larger context window, Gemini can process entire documents instead of just smaller chunks.\n",
        "\n",
        "## **Why Use a Large Context Window?**\n",
        "\n",
        "  * More Comprehensive Answers: By seeing the full context of a document, Gemini can provide more nuanced and detailed answers to your questions.\n",
        "\n",
        "  * Better Understanding: The model gains a better understanding of the relationships and dependencies within the document, leading to more accurate responses.\n",
        "\n",
        "  * Reduced Fragmentation: You can avoid potential issues with information being fragmented across different chunks, ensuring a smoother flow of information.\n",
        "\n",
        "\n",
        "## **How it Works:**\n",
        "\n",
        "**1. Load the Document:**\n",
        "  * Select the loan document you want to use (from Google Cloud Storage or your local files).\n",
        "\n",
        "**2. Prepare the Input:**\n",
        "\n",
        "* The system reads the entire document and formats it for input into the Gemini model.\n",
        "\n",
        "**3. Ask Your Question:**\n",
        "\n",
        "* Pose your question related to the document.\n",
        "\n",
        "**4. Generate Answer:**\n",
        "\n",
        "* Gemini processes the entire document along with your question to generate a response. This response is informed by the full context of the document, leading to more in-depth answers.\n",
        "\n",
        "\n",
        "## **Benefits:**\n",
        "\n",
        "* Enhanced Detail: Get more comprehensive answers that reflect the full content of the document.\n",
        "* Improved Accuracy: Reduce the chances of misinterpretations by providing the model with complete context.\n",
        "* Greater Flexibility: Explore complex questions that require understanding information across different sections of the document.\n",
        "\n",
        "\n",
        "## **Example:**\n",
        "\n",
        "Let's say you want to know about various loan types and their specific details. By using a large context window, Gemini can analyze the entire loan agreement document and provide you with a more comprehensive answer, including specific clauses and conditions related to each type of loan.\n",
        "\n",
        "## **Limitations:**\n",
        "  * Processing Time: Working with large documents can take longer compared to using smaller chunks.\n",
        "  * Resource Requirements: It may require more computing power and memory depending on the document size.\n",
        "\n",
        "## **When to Use It:**\n",
        "\n",
        "Consider using a large context window when:\n",
        "\n",
        "  * You need deeper insights and comprehensive answers.\n",
        "  * Your questions involve understanding information spread across the entire document.\n",
        "  * Accuracy and detailed analysis are critical for your task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hIfczQlbhG4y"
      },
      "outputs": [],
      "source": [
        "# Taking document from the GCS path\n",
        "# document_path = \"gs://github-repo/generative-ai/gemini2/use-cases/loan_example_documents/Demo-bank-home-loan-agreement.pdf\"\n",
        "# document_content = extract_text_from_gcs(document_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e7GNNxptVVHk"
      },
      "outputs": [],
      "source": [
        "# Taking document from the local path\n",
        "# download the example file and keep in the Colab files.\n",
        "document_path = \"/content/DEMO-BANK-LOAN-DETAILS.pdf\"\n",
        "document_content = extract_text_from_pdf(document_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7wuk15mxEfxE"
      },
      "source": [
        "### Text Output - Large Context"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Teg4LiNqEPxh"
      },
      "outputs": [],
      "source": [
        "query = \"what are the type of loans?\"\n",
        "\n",
        "large_context_prompt = f\"\"\"Based on the following context, please answer the question.\n",
        "\n",
        "  Context:\n",
        "  {document_content}\n",
        "\n",
        "  Question: {query}\n",
        "\n",
        "  Answer:\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ogwQmErQPb-g"
      },
      "outputs": [],
      "source": [
        "response = client.models.generate_content(model=MODEL, contents=large_context_prompt)\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JggCQgsdv16Q"
      },
      "source": [
        "### Text In Audio Out, Multimodal Live API"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rLjZeApGvy4d"
      },
      "outputs": [],
      "source": [
        "await generate_n_play_audio(client, large_context_prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ZShwrOt0KSi"
      },
      "source": [
        "## Option 3 : With Vertex AI Search\n",
        "\n",
        "<div class=\"alert alert-block alert-warning\">\n",
        "\n",
        "<b>⚠️ Assumption, you have already created or have a datastore with your documents, If not please follow this guide to create it <a href=\"https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-es#cloud-storage\"> Create a search datastore</a> ⚠️</b>\n",
        "</div>\n",
        "\n",
        "This section demonstrates how to use Gemini with Vertex AI Search for question answering. Vertex AI Search is a fully managed, scalable NoSQL database service that can store and retrieve large amounts of data.\n",
        "\n",
        "- What is Vertex AI Search?\n",
        "\n",
        "  - Vertex AI Search is a database service that can store and retrieve large amounts of data. It is fully managed and scalable, making it a good choice for storing large datasets. In this context, it is used to store the loan documents and their embeddings, which are used by Gemini to answer questions about the documents.\n",
        "\n",
        "- Why Use Vertex AI Search?\n",
        "\n",
        "  - Scalability: Vertex AI Search can handle large datasets and high traffic loads.\n",
        "  - Reliability: Vertex AI Search is a fully managed service, so you don't have to worry about managing the infrastructure.\n",
        "  - Integration: Vertex AI Search integrates well with other Google Cloud services, such as Gemini.\n",
        "\n",
        "- How it Works:\n",
        "\n",
        "  1. Store the Documents: The loan documents and their embeddings are stored in Vertex AI Search.\n",
        "  2. Ask Your Question: You ask a question about the documents.\n",
        "  3. Retrieve Relevant Information: Gemini uses the embeddings to quickly identify the most relevant documents in Vertex AI Search.\n",
        "  4. Generate Answer: Gemini processes the relevant documents along with your question to generate a response.\n",
        "\n",
        "- Benefits\n",
        "\n",
        "  - Efficiency: Vertex AI Search enables fast and efficient retrieval of relevant information.\n",
        "  - Scalability: You can easily handle large datasets and high traffic loads.\n",
        "  - Integration: It seamlessly integrates with Gemini and other Google Cloud services.\n",
        "\n",
        "### Example\n",
        "\n",
        "Let's say you want to know about the different types of loans available. Using Vertex AI Search, you can simply ask the question, and Gemini will:\n",
        "\n",
        "1. Identify the relevant documents in Vertex AI Search.\n",
        "2. Use Gemini to extract and generate an answer based on those documents.\n",
        "3. Present the answer to you in a clear and concise format.\n",
        "\n",
        "### Limitations:\n",
        "\n",
        "* Cost: Vertex AI Search is a paid service, so you will need to pay for storage and retrieval costs.\n",
        "\n",
        "### When to Use It:\n",
        "\n",
        "Consider using Vertex AI Search when:\n",
        "\n",
        "* You don't want to manage the RAG infrastructure and complex logic\n",
        "* You have a large dataset of documents.\n",
        "* You need high performance and scalability.\n",
        "* You want to integrate with other Google Cloud services."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SSHYTbIq5Q7N"
      },
      "source": [
        "### Initialize the Vertex AI Search data store"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-aH8OrA_0VMr"
      },
      "outputs": [],
      "source": [
        "## Vertex AI Search\n",
        "\n",
        "datastore_id = \"Your-datastore\"  # @param {type: \"string\", isTemplate: true}\n",
        "\n",
        "datastore_path = f\"projects/{PROJECT_ID}/locations/global/collections/default_collection/dataStores/{datastore_id}\"\n",
        "\n",
        "vertex_ai_search_tool = Tool(\n",
        "    retrieval=Retrieval(vertex_ai_search=VertexAISearch(datastore=datastore_path))\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EU__4wc65Zhi"
      },
      "source": [
        "### Get your QnA with Vertex AI Search and Vertex AI Multimodal Live API\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MT_SK9iC0ZnF"
      },
      "outputs": [],
      "source": [
        "query = \"what are the types of loans?\"  # @param  {type: \"string\", isTemplate: true, label: \"Select Modality\"}\n",
        "\n",
        "modality = \"TEXT\"\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=query,\n",
        "    config=GenerateContentConfig(\n",
        "        tools=[vertex_ai_search_tool], response_modalities=[modality]\n",
        "    ),\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iiEE2Brm5MkU"
      },
      "source": [
        "## Option 4: with Vertex AI Vector Search\n",
        "\n",
        "This section demonstrates how to use Gemini with Vertex AI Vector Search for question answering. Vertex AI Vector Search is a fully managed, scalable search service that can be used to power retrieval augmented generation (RAG) applications.\n",
        "\n",
        "- What is Vertex AI Vector Search?\n",
        "\n",
        "  - Vertex AI Vector Search is a powerful search service that allows you to build applications that can understand and respond to natural language queries. It uses advanced machine learning techniques to index and search your data, and it can be used to power a variety of applications, including RAG.\n",
        "\n",
        "- Why Use Vertex AI Vector Search?\n",
        "\n",
        "  - Scalability: Vertex AI Vector Search can handle large datasets and high traffic loads. This means you can use it to build applications that can scale to meet the needs of your business.\n",
        "  - Reliability: Vertex AI Vector Search is a fully managed service, so you don't have to worry about managing the infrastructure. This means you can focus on building your application, without having to worry about the underlying infrastructure.\n",
        "  - Integration: Vertex AI Vector Search integrates well with other Google Cloud services, such as Gemini. This makes it easy to build applications that use multiple Google Cloud services.\n",
        "\n",
        "- How it Works:\n",
        "\n",
        "  1. Create a Search Index: You first need to create a search index for your data. This is done using the Vertex AI Vector Search API.\n",
        "  2. Ask Your Question: You ask a question about your data.\n",
        "  3. Retrieve Relevant Information: Vertex AI Vector Search uses its index to quickly identify the most relevant information to your query.\n",
        "  4. Generate Answer: Gemini processes the relevant information along with your question to generate a response.\n",
        "\n",
        "- Benefits\n",
        "\n",
        "  - Efficiency: Vertex AI Vector Search enables fast and efficient retrieval of relevant information.\n",
        "  - Scalability: You can easily handle large datasets and high traffic loads.\n",
        "  - Integration: It seamlessly integrates with Gemini and other Google Cloud services.\n",
        "\n",
        "### Example\n",
        "\n",
        "Let's say you want to know about the different types of loans available. Using Vertex AI Vector Search, you can simply ask the question, and Gemini will:\n",
        "\n",
        "1. Identify the relevant information in your search index.\n",
        "2. Use Gemini to extract and generate an answer based on that information.\n",
        "3. Present the answer to you in a clear and concise format.\n",
        "\n",
        "### Limitations\n",
        "\n",
        "Cost: Vertex AI Vector Search is a paid service, so you will need to pay for storage and retrieval costs.\n",
        "\n",
        "### When to Use It\n",
        "\n",
        "Consider using Vertex AI Vector Search when:\n",
        "\n",
        "* You have a large dataset of documents.\n",
        "* You need high performance and scalability.\n",
        "* You want to integrate with other Google Cloud services.\n",
        "* You want to improve search result diversity, quality, and ranking through ranking and recall tuning features of vector search."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zc14X-p31sci"
      },
      "source": [
        "### Import for Vertex AI"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l_uQrCum1U2O"
      },
      "outputs": [],
      "source": [
        "from google.cloud import aiplatform\n",
        "\n",
        "aiplatform.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etYrU8Zq1xEC"
      },
      "source": [
        "### Setup Vertex AI Vector Search index and index endpoint\n",
        "\n",
        "In this section, we have some helper methods to help you setup your Vector Search index.\n",
        "\n",
        "This section is not required if you already have a Vector Search index ready to use.\n",
        "\n",
        "The index has to meet the following criteria:\n",
        "\n",
        "1. `IndexUpdateMethod` must be `STREAM_UPDATE`, see [Create stream index]({{docs_path}}vector-search/create-manage-index#create-stream-index).\n",
        "\n",
        "2. Distance measure type must be explicitly set to one of the following:\n",
        "\n",
        "   * `DOT_PRODUCT_DISTANCE`\n",
        "   * `COSINE_DISTANCE`\n",
        "\n",
        "3. Dimension of the vector must be consistent with the embedding model you plan\n",
        "   to use in the RAG corpus. Other parameters can be tuned based on\n",
        "   your choices, which determine whether the additional parameters can be\n",
        "   tuned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1M_dnU9P1gvQ"
      },
      "outputs": [],
      "source": [
        "# create the index\n",
        "my_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(\n",
        "    display_name=\"loanDemoRag\",\n",
        "    description=\"Demo app for rag with banking use case\",\n",
        "    dimensions=768,\n",
        "    approximate_neighbors_count=10,\n",
        "    leaf_node_embedding_count=500,\n",
        "    leaf_nodes_to_search_percent=7,\n",
        "    distance_measure_type=\"DOT_PRODUCT_DISTANCE\",\n",
        "    feature_norm_type=\"UNIT_L2_NORM\",\n",
        "    index_update_method=\"STREAM_UPDATE\",\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gnPusIuK1mpD"
      },
      "source": [
        "### Vertex ai search public endpoint [public endpoints](https://cloud.google.com/vertex-ai/docs/vector-search/deploy-index-public)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "If92NcCT1mDD"
      },
      "outputs": [],
      "source": [
        "# create IndexEndpoint\n",
        "my_index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(\n",
        "    display_name=\"loanDemoRag\", public_endpoint_enabled=True\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0VCAa0qe2X6F"
      },
      "source": [
        "Deploying the Index to the Index Endpoint\n",
        "When deploying an index to an index endpoint for the first time, it takes approximately 30 minutes to automatically build and initialize the backend. Subsequent deployments are significantly faster, with the index becoming ready in seconds.\n",
        "\n",
        "To monitor the status of the index deployment:\n",
        "\n",
        "Open the Vector Search Console.\n",
        "Navigate to the Index endpoints tab.\n",
        "Select your index endpoint to view its details.\n",
        "Resource Name Formats\n",
        "Ensure you have the correct resource names for your index and index endpoint in the following formats:\n",
        "\n",
        "* `projects/${PROJECT_ID}/locations/${LOCATION_ID}/indexes/${INDEX_ID}`\n",
        "* `projects/${PROJECT_ID}/locations/${LOCATION_ID}/indexEndpoints/${INDEX_ENDPOINT_ID}`.\n",
        "\n",
        "\n",
        "If you're unsure of the resource names, use the following command to retrieve them:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CAB9Sj0p2YUH"
      },
      "outputs": [],
      "source": [
        "print(my_index_endpoint.resource_name)\n",
        "print(my_index.resource_name)\n",
        "print(my_index.name)\n",
        "print(my_index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IteN67ln20cm"
      },
      "outputs": [],
      "source": [
        "# Deploy Index\n",
        "my_index_endpoint.deploy_index(index=my_index, deployed_index_id=\"loanDemoRag\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JqggwGu721AT"
      },
      "outputs": [],
      "source": [
        "from vertexai.preview import rag"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "caa_Vprs237T"
      },
      "outputs": [],
      "source": [
        "vector_db = rag.VertexVectorSearch(\n",
        "    index=my_index.resource_name, index_endpoint=my_index_endpoint.resource_name\n",
        ")\n",
        "\n",
        "# Name your corpus\n",
        "DISPLAY_NAME = \"loan_app_corpus\"  # @param  {type:\"string\"}\n",
        "\n",
        "# Create RAG Corpus\n",
        "rag_corpus = rag.create_corpus(display_name=DISPLAY_NAME, vector_db=vector_db)\n",
        "print(f\"Created RAG Corpus resource: {rag_corpus.name}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XuTDZP7c28yi"
      },
      "source": [
        "### Import the files from the GCS"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_V_S8aC_29Mj"
      },
      "outputs": [],
      "source": [
        "GCS_BUCKET = \"gs://demo-loan-documents/\"  # @param {type:\"string\", \"placeholder\": \"your-gs-bucket\"}\n",
        "\n",
        "response = rag.import_files(  # noqa: F704\n",
        "    corpus_name=rag_corpus.name,\n",
        "    paths=[GCS_BUCKET],\n",
        "    chunk_size=512,\n",
        "    chunk_overlap=50,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8U_TcPZU3C0j"
      },
      "outputs": [],
      "source": [
        "### Check the files just imported. It may take a few seconds to process the imported files.\n",
        "rag.list_files(corpus_name=rag_corpus.name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HCpBI2DZ3F_Z"
      },
      "source": [
        "### Add Rag corpus to the context"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jwjcsMjx3Gem"
      },
      "outputs": [],
      "source": [
        "rag_resource = rag.RagResource(\n",
        "    rag_corpus=rag_corpus.name,\n",
        ")\n",
        "\n",
        "vertex_ai_rag_tool = Tool(\n",
        "    retrieval=Retrieval(\n",
        "        vertex_rag_store=VertexRagStore(\n",
        "            rag_resources=[rag_resource],  # Currently only 1 corpus is allowed.\n",
        "            similarity_top_k=10,\n",
        "            vector_distance_threshold=0.4,\n",
        "        ),\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jqljXg1f3VzU"
      },
      "outputs": [],
      "source": [
        "query = \"what are the types of loans?\"  # @param  {type: \"string\", isTemplate: true, label: \"Select Modality\"}\n",
        "\n",
        "modality = \"TEXT\"\n",
        "\n",
        "\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=query,\n",
        "    config=GenerateContentConfig(\n",
        "        tools=[vertex_ai_rag_tool], response_modalities=[modality]\n",
        "    ),\n",
        ")\n",
        "\n",
        "display(Markdown(response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4PWWqy4J_lTm"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "In this tutorial, you have learned how to leverage the capabilities of Gemini 2.0, covering the following topics:\n",
        "\n",
        "* Using the Google Gen AI SDK: Learn how to interact with Gemini models through the SDK.\n",
        "\n",
        "* Utilizing the Google Multimodal Live API: Explore its features for handling multimodal data.\n",
        "\n",
        "* Creating and Implementing Retrieval Augmented Generation (RAG): Define and apply RAG techniques effectively.\n",
        "\n",
        "* Extracting Information from PDFs:\n",
        "    * Converting Text to Text.\n",
        "    * Generating Text to Audio.\n",
        "* Leveraging a Large Context Window:\n",
        "\n",
        "    * Maximize capabilities with the Gen AI SDK and Multimodal Live API.\n",
        "\n",
        "* Working with Vertex AI Features:\n",
        "    * Vertex AI Search for efficient data management.\n",
        "    * Vertex AI Search for enhanced query capabilities.\n",
        "\n",
        "* Developing Q&A Applications: Build a Question-and-Answer application with Gemini 2.0.\n",
        "\n",
        "This comprehensive guide equips you with practical knowledge for utilizing Gemini 2.0 in diverse scenarios, from multimodal data handling to advanced AI-powered application development.\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "real_time_rag_bank_loans_gemini_2_0.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
