{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bCIMTPB1WoTq"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7yVV6txOmNMn"
      },
      "source": [
        "# Basic prototyping of the solution with Gemini API in Vertex AI\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/1_prototyping_gemini.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fworkshops%2Frag-ops%2F1_prototyping_gemini.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/workshops/rag-ops/1_prototyping_gemini.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/1_prototyping_gemini.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/1_prototyping_gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/1_prototyping_gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/1_prototyping_gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/1_prototyping_gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/1_prototyping_gemini.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t1DnOs6rkbOy"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Gemini 2.0 is a new language model from the Gemini family. This model introduces a breakthrough long context window of up to 1 million tokens that can help seamlessly analyze large amounts of information and long-context understanding. It can process text, images, audio, video, and code all together for deeper insights. Learn more about [Gemini 2.0](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/).\n",
        "\n",
        "With this tutorial, you learn how to use the Gemini API in Vertex AI and the Vertex AI SDK to work with the Gemini 2.0 model to:\n",
        "\n",
        "- analyze audio for insights.\n",
        "- understand videos (including their audio components).\n",
        "- extract information from PDF documents.\n",
        "- process images, video, audio, and text simultaneously."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK for Python\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user --quiet google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the cell below to authenticate your environment.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "import vertexai\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\", isTemplate: true}\n",
        "\n",
        "if PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\" or PROJECT_ID == \"None\":\n",
        "    raise ValueError(\"Please set your PROJECT_ID\")\n",
        "\n",
        "print(f\"Project ID: {PROJECT_ID}\")\n",
        "\n",
        "LOCATION = \"us-central1\"\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jXHfaVS66_01"
      },
      "source": [
        "### Import libraries\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lslYAvw37JGQ"
      },
      "outputs": [],
      "source": [
        "import IPython.display\n",
        "from IPython.core.interactiveshell import InteractiveShell\n",
        "\n",
        "InteractiveShell.ast_node_interactivity = \"all\"\n",
        "\n",
        "from vertexai.generative_models import (\n",
        "    GenerationConfig,\n",
        "    GenerativeModel,\n",
        "    HarmBlockThreshold,\n",
        "    HarmCategory,\n",
        "    Part,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BY1nfXrqRxVX"
      },
      "source": [
        "### Load the Gemini 2.0 model\n",
        "\n",
        "To learn more about all [Gemini API models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models).\n",
        "\n",
        "The Gemini model family has several model versions. You will start by using Gemini 2.0. Gemini 2.0 is a more lightweight, fast, and cost-efficient model. This makes it a great option for prototyping.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U7ExWmuLBdIA"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.0-flash\"  # @param {type:\"string\"}\n",
        "\n",
        "model = GenerativeModel(MODEL_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l9OKM0-4SQf8"
      },
      "source": [
        "## Vertex AI SDK basic usage\n",
        "\n",
        "Prototyping starts with calling the API to see if we get a response. Lets learn some of the funamentals of the Vertex AI Gemini model.\n",
        "\n",
        "Below is a simple example that demonstrates how to prompt the Gemini 2.0 model using the Vertex AI SDK. Learn more about the [Gemini API parameters](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini#gemini-pro)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FhFxrtfdSwOP"
      },
      "outputs": [],
      "source": [
        "%time\n",
        "\n",
        "# Load an example model with system instructions\n",
        "example_model = GenerativeModel(\n",
        "    MODEL_ID,\n",
        "    system_instruction=[\n",
        "        \"You are a financial analyst specialized in earnings reports\",\n",
        "    ],\n",
        ")\n",
        "\n",
        "# Set model parameters\n",
        "generation_config = GenerationConfig(\n",
        "    temperature=0.9,\n",
        "    top_p=1.0,\n",
        "    top_k=32,\n",
        "    candidate_count=1,\n",
        "    max_output_tokens=8192,\n",
        ")\n",
        "\n",
        "# Set safety settings\n",
        "safety_settings = {\n",
        "    HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
        "    HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
        "    HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
        "    HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
        "}\n",
        "\n",
        "prompt = \"\"\"\n",
        "  User input: Explain the steps you would take to review a companies earnings report?\n",
        "  Answer:\n",
        "\"\"\"\n",
        "\n",
        "# Set contents to send to the model\n",
        "contents = [prompt]\n",
        "\n",
        "# Counts tokens\n",
        "print(example_model.count_tokens(contents))\n",
        "\n",
        "# Prompt the model to generate content\n",
        "response = example_model.generate_content(\n",
        "    contents,\n",
        "    generation_config=generation_config,\n",
        "    safety_settings=safety_settings,\n",
        ")\n",
        "\n",
        "# Print the model response\n",
        "print(f\"\\nAnswer:\\n{response.text}\")\n",
        "print(f\"\\nUsage metadata:\\n{response.to_dict().get('usage_metadata')}\")\n",
        "print(f\"\\nFinish reason:\\n{response.candidates[0].finish_reason}\")\n",
        "print(f\"\\nSafety settings:\\n{response.candidates[0].safety_ratings}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1yhiPa46BGV0"
      },
      "source": [
        "### Enable streaming\n",
        "\n",
        "You can also enable streaming to receive the output while its being generated."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y6H9DYnwBFmN"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "response = example_model.generate_content(\n",
        "    contents,\n",
        "    stream=True,\n",
        "    generation_config=generation_config,\n",
        "    safety_settings=safety_settings,\n",
        ")\n",
        "\n",
        "for chunk in response:\n",
        "    print(chunk.text)\n",
        "    print(\"_\" * 80)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6pAX8PQ0BhC2"
      },
      "source": [
        "### Token count\n",
        "**Important:** A token is equivalent to about 4 characters for Gemini models. 100 tokens are about 60-80 English words.\n",
        "[Cloud pricing](https://cloud.google.com/vertex-ai/generative-ai/pricing) is done on characters and not tokens.\n",
        "\n",
        "Using token count can help you with understanding the cost of running your prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2EOeT5--CBTa"
      },
      "outputs": [],
      "source": [
        "# Get token count\n",
        "response = model.count_tokens(prompt)\n",
        "print(f\"Prompt Token Count: {response.total_tokens}\")\n",
        "print(f\"Prompt Character Count: {response.total_billable_characters}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ElUrciMe9_Z9"
      },
      "source": [
        "### Prompt design fundamentals\n",
        "Here are some fundamentals for designing your prompts throughout this workshop:\n",
        "- Be specific in your instructions: Craft clear and concise instructions that leave minimal room for misinterpretation.\n",
        "- - Add a few examples to your prompt: Use realistic few-shot examples to illustrate what you want to achieve.\n",
        "Break it down step-by-step: Divide complex tasks into manageable sub-goals, guiding the model through the process.\n",
        "- Specify the output format: In your prompt, ask for the output to be in the format you want, like markdown, JSON, HTML and more.\n",
        "- Put your image first for single-image prompts: While Gemini can handle image and text inputs in any order, for prompts containing a single image, it might perform better if that image (or video) is placed before the text prompt. However, for prompts that require images to be highly interleaved with texts to make sense, use whatever order is most natural."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8jpFgHqCvQE"
      },
      "source": [
        "## Working with multimodal data\n",
        "\n",
        "Lets now dive into the world of multimodal where we will use different types of data, like a pdf and audio, and ask Gemini to reasons across these modalities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FL-wDJCrRHZQ"
      },
      "source": [
        "### Text and PDF\n",
        "\n",
        "Next, you can ask a question about the PDF document. You will use the PDF format of an earnings report. To answer this question the model has to process the PDF and find the information in the pdf."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aFnAXw31yoj3"
      },
      "outputs": [],
      "source": [
        "ROLE = \"\"\"\n",
        "You are a financial analyst. You specialize in Give me a summary of the earnings report\n",
        "Only base your answer on the information provided.\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jCP2T-fe2jzb"
      },
      "outputs": [],
      "source": [
        "question_1 = \"Question: How many shares of each class of Alphabet stock were outstanding as of July 22, 2022?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4JS7PgyI3HAH"
      },
      "source": [
        "Next you can combine the prompts with the PDF and send it to the Gemini API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EKYRfscF5_q1"
      },
      "outputs": [],
      "source": [
        "pdf_file_uri = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/goog-10-q-q2-2023-4-1-15.pdf\"\n",
        "pdf_file = Part.from_uri(pdf_file_uri, mime_type=\"application/pdf\")\n",
        "contents = [pdf_file, ROLE, question_1]\n",
        "\n",
        "response = model.generate_content(contents)\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rn6q4c0V5_vb"
      },
      "outputs": [],
      "source": [
        "question_2 = (\n",
        "    \"Summarize the earnings call provided and provide us with your top 3 take aways\"\n",
        ")\n",
        "\n",
        "audio_file_uri = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/alphabet_2023_q4_earnings_call.mp3\"\n",
        "audio_file = Part.from_uri(audio_file_uri, mime_type=\"audio/mpeg\")\n",
        "\n",
        "contents = [audio_file, ROLE, question_2]\n",
        "\n",
        "response = model.generate_content(contents)\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xqO6ylxxDDMV"
      },
      "source": [
        "### Troubleshooting your multimodal prompt\n",
        "While prototyping you also might have to troubleshoot your multimodal prompts. Here are some tips:\n",
        "\n",
        "- If the model is not drawing information from the relevant part of the image: Drop hints with which aspects of the image you want the prompt to draw information from.\n",
        "- If the model output is too generic (not tailored enough to the image/video input): At the start of the prompt, try asking the model to describe the image(s) or video before providing the task instruction, or try asking the model to refer to what's in the image.\n",
        "- To troubleshoot which part failed: Ask the model to describe the image, or ask the model to explain its reasoning, to gauge the model's initial understanding.\n",
        "- If your prompt results in hallucinated content: Try dialing down the temperature setting or asking the model for shorter descriptions so that it's less likely to extrapolate additional details.\n",
        "- Tuning the sampling parameters: Experiment with different temperature settings and top-k selections to adjust the model's creativity."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2amIRb-OSNM5"
      },
      "source": [
        "## Advanced reasoning across text, audio, video and PDF.\n",
        "\n",
        "Let's do something more complex. You will now let the model reason across different modalities to find the answer to the question: audio, PDF, and video.\n",
        "\n",
        "Lets also switch models and use `gemini-2.0-flash`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n002WnWNbfYa"
      },
      "outputs": [],
      "source": [
        "MODEL_ID = \"gemini-2.0-flash\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8P_L2E4ROYTg"
      },
      "outputs": [],
      "source": [
        "ROLE = \"You are a financial analyst at a hedge fund\"\n",
        "question_2 = \"How is Google using AI to optimize advertisement experience and what strategic partnership did Google announce in February 2023 in the automotive industry??\"\n",
        "\n",
        "audio_uri_1 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/Alphabet_2023_Q1_Earnings_Call.mp3\"\n",
        "audio_uri_2 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/episode1.mp3\"\n",
        "\n",
        "pdf_uri_2 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/20230426-alphabet-10q.pdf\"\n",
        "video_uri_1 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/changing-the-way-scientists-research-Gemini.mp4\"\n",
        "\n",
        "pdf_file = Part.from_uri(pdf_uri_2, mime_type=\"application/pdf\")\n",
        "video_file = Part.from_uri(video_uri_1, mime_type=\"video/mp4\")\n",
        "audio_file_1 = Part.from_uri(audio_uri_1, mime_type=\"audio/mpeg\")\n",
        "audio_file_2 = Part.from_uri(audio_uri_2, mime_type=\"audio/mpeg\")\n",
        "\n",
        "contents = [audio_file_1, audio_file_2, video_file, pdf_file, ROLE, question_2]\n",
        "\n",
        "response = model.generate_content(contents)\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P_6z78-jd21R"
      },
      "source": [
        "## Experimentation\n",
        "\n",
        "Now its time for you to improve the prompts. You can update the prompts below and re-run the code to see the response from Gemini."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J_7nr1I3eSfK"
      },
      "outputs": [],
      "source": [
        "ROLE = \"<update-this>\"\n",
        "question_2 = \"<update-this>\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4ENMZk0rd4q9"
      },
      "outputs": [],
      "source": [
        "audio_uri_1 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/Alphabet_2023_Q1_Earnings_Call.mp3\"\n",
        "audio_uri_2 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/episode1.mp3\"\n",
        "\n",
        "pdf_uri_2 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/20230426-alphabet-10q.pdf\"\n",
        "video_uri_1 = \"gs://mlops-for-genai/multimodal-finanace-qa/data/unstructured/prototype/changing-the-way-scientists-research-Gemini.mp4\"\n",
        "\n",
        "pdf_file = Part.from_uri(pdf_uri_2, mime_type=\"application/pdf\")\n",
        "video_file = Part.from_uri(video_uri_1, mime_type=\"video/mp4\")\n",
        "audio_file_1 = Part.from_uri(audio_uri_1, mime_type=\"audio/mpeg\")\n",
        "audio_file_2 = Part.from_uri(audio_uri_2, mime_type=\"audio/mpeg\")\n",
        "\n",
        "# contents = [audio_file_1, audio_file_2, video_file, pdf_file, ROLE, question_2]\n",
        "contents = [audio_file_1, audio_file_2, ROLE, question_2]\n",
        "\n",
        "response = model.generate_content(contents)\n",
        "print(response.text)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "1_prototyping_gemini.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
