{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# YouTube Video Analysis with Gemini\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Fvideo-analysis%2Fyoutube_video_analysis.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/video-analysis/youtube_video_analysis.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84f0f73a0f76"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "| Author(s) | [Alok Pattani](https://github.com/alokpattani/) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "In this notebook, you'll explore how to do direct analysis of publicly available [YouTube](https://www.youtube.com/) videos with Gemini.\n",
        "\n",
        "You will complete the following tasks:\n",
        "\n",
        "- Summarizing a single YouTube video using Gemini 2.0 Flash\n",
        "- Extracting a specific set of structured outputs from a longer YouTube video using Gemini 2.0 Pro and controlled generation\n",
        "- Creating insights from analyzing multiple YouTube videos together using asynchronous generation with Gemini"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Google Gen AI SDK and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-genai itables"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
        "\n",
        "The restart might take a minute or longer. After it's restarted, continue to the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information and create client\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "# Use the environment variable if the user doesn't provide Project ID.\n",
        "import os\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
        "\n",
        "from google import genai\n",
        "\n",
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdvJRUWRNGHE"
      },
      "source": [
        "## Set up libraries, options, and models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5303c05f7aa6"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "6fc324893334"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "from IPython.display import HTML, Markdown, display\n",
        "from google.genai.types import GenerateContentConfig, Part\n",
        "from itables import show\n",
        "import itables.options as itable_opts\n",
        "import pandas as pd\n",
        "from tenacity import retry, stop_after_attempt, wait_random_exponential"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f86c665a5d94"
      },
      "source": [
        "### Configure some notebook options"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "4730b9f09e1e"
      },
      "outputs": [],
      "source": [
        "# Configure some options related to interactive tables\n",
        "itable_opts.maxBytes = 1e9\n",
        "itable_opts.maxColumns = 50\n",
        "\n",
        "itable_opts.order = []\n",
        "itable_opts.column_filters = \"header\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9d46fd0dfdf7"
      },
      "source": [
        "### Create a helper function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "e9034b0991f4"
      },
      "outputs": [],
      "source": [
        "def display_youtube_video(url: str) -> None:\n",
        "    youtube_video_embed_url = url.replace(\"/watch?v=\", \"/embed/\")\n",
        "\n",
        "    # Create HTML code to directly embed video\n",
        "    youtube_video_embed_html_code = f\"\"\"\n",
        "    <iframe width=\"560\" height=\"315\" src=\"{youtube_video_embed_url}\"\n",
        "    title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; \n",
        "    clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen>\n",
        "    </iframe>\n",
        "    \"\"\"\n",
        "\n",
        "    # Display embedded YouTube video\n",
        "    display(HTML(youtube_video_embed_html_code))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e43229f3ad4f"
      },
      "source": [
        "### Load models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "cf93d5f0ce00"
      },
      "outputs": [],
      "source": [
        "# Set Gemini Flash and Pro models to be used in this notebook\n",
        "GEMINI_FLASH_MODEL_ID = \"gemini-2.0-flash-001\"\n",
        "GEMINI_PRO_MODEL_ID = \"gemini-2.0-flash\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "109111fae02c"
      },
      "source": [
        "## Summarize a YouTube video"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1fe7e2663c3a"
      },
      "source": [
        "Provide a link to a public YouTube video that you'd like to summarize. Ensure that the video is less than an hour long (if using Gemini 2.0, as is shown below; can try up to a 2-hour video with Gemini 2.0) to make sure it fits in the context window.\n",
        "\n",
        "The default content to be summarized is [this 6.5-minute video showing how Major League Baseball (MLB) analyzes data using Google Cloud](https://www.youtube.com/watch?v=O_W_VGUeHVI)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5c8a32e14eec"
      },
      "outputs": [],
      "source": [
        "# Provide link to a public YouTube video to summarize\n",
        "YOUTUBE_VIDEO_URL = (\n",
        "    \"https://www.youtube.com/watch?v=O_W_VGUeHVI\"  # @param {type:\"string\"}\n",
        ")\n",
        "\n",
        "display_youtube_video(YOUTUBE_VIDEO_URL)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9bd742163fc7"
      },
      "outputs": [],
      "source": [
        "# Call Gemini API with prompt to summarize video\n",
        "video_summary_prompt = \"Give a detailed summary of this video.\"\n",
        "\n",
        "video_summary_response = client.models.generate_content(\n",
        "    model=GEMINI_FLASH_MODEL_ID,\n",
        "    contents=[\n",
        "        Part.from_uri(\n",
        "            file_uri=YOUTUBE_VIDEO_URL,\n",
        "            mime_type=\"video/webm\",\n",
        "        ),\n",
        "        video_summary_prompt,\n",
        "    ],\n",
        ")\n",
        "\n",
        "# Display results\n",
        "display(Markdown(video_summary_response.text))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "09221a4ba6a9"
      },
      "source": [
        "## Extract structured output from a YouTube video"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "db6bc26fca7d"
      },
      "source": [
        "Next, we'll show how to extract structured outputs using [controlled generation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output), in this case from a video that covers multiple topics.\n",
        "\n",
        "We're going to see how Gemini's industry-leading 2 million token context window can help analyze [the full opening keynote](https://www.youtube.com/watch?v=V6DJYGn2SFk) from our Next conference back in April - all 1 hour and 41 minutes of it!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fc98b36d5fc4"
      },
      "outputs": [],
      "source": [
        "# Link to full Cloud Next '24 Opening Keynote video\n",
        "# cloud_next_keynote_video_url = \"https://www.youtube.com/watch?v=V6DJYGn2SFk\"\n",
        "\n",
        "# Uncomment line below to replace with 14-min keynote summary video instead (faster)\n",
        "cloud_next_keynote_video_url = \"https://www.youtube.com/watch?v=M-CzbTUVykg\"\n",
        "\n",
        "display_youtube_video(cloud_next_keynote_video_url)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e904c020d521"
      },
      "source": [
        "Below is a prompt to extract the biggest product announcements that were made during this keynote. We use the response schema to show that we want valid JSON output in a particular form, including a constraint specifying that the \"product status\" field should be either GA, Preview, or Coming Soon.\n",
        "\n",
        "The following cell may take several minutes to run, as Gemini 2.0 Pro is analyzing all 101 minutes of the video and audio to produce comprehensive results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d5a93cd5d2fa"
      },
      "outputs": [],
      "source": [
        "# Set up pieces (prompt, response schema, config) and run video extraction\n",
        "\n",
        "video_extraction_prompt = (\n",
        "    \"Provide a summary of the biggest product announcements \"\n",
        "    \"that were made in this Google Cloud Next keynote video including:\\n\"\n",
        "    \"  - name\\n\"\n",
        "    '  - product status: \"GA\" (Generally Available), \"Preview\", or \"Coming Soon\"\\n'\n",
        "    \"  - key quote from the presenter about the product, 20 words or fewer per product\\n\\n\"\n",
        "    \"Make sure to look through and listen to the whole video, start to finish, to find \"\n",
        "    \"the top product announcements. Only reference information in the video itself in \"\n",
        "    \"your response.\"\n",
        ")\n",
        "\n",
        "video_extraction_response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"OBJECT\",\n",
        "        \"properties\": {\n",
        "            \"name\": {\"type\": \"STRING\"},\n",
        "            \"product_status\": {\n",
        "                \"type\": \"STRING\",\n",
        "                \"enum\": [\"GA\", \"Preview\", \"Coming Soon\"],\n",
        "            },\n",
        "            \"quote_from_presenter\": {\"type\": \"STRING\"},\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "video_extraction_json_generation_config = GenerateContentConfig(\n",
        "    temperature=0.0,\n",
        "    max_output_tokens=8192,\n",
        "    response_mime_type=\"application/json\",\n",
        "    response_schema=video_extraction_response_schema,\n",
        ")\n",
        "\n",
        "video_extraction_response = client.models.generate_content(\n",
        "    model=GEMINI_PRO_MODEL_ID,\n",
        "    contents=[\n",
        "        video_extraction_prompt,\n",
        "        Part.from_uri(\n",
        "            file_uri=cloud_next_keynote_video_url,\n",
        "            mime_type=\"video/webm\",\n",
        "        ),\n",
        "    ],\n",
        "    config=video_extraction_json_generation_config,\n",
        ")\n",
        "\n",
        "print(video_extraction_response.text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b7b6aa978eb8"
      },
      "outputs": [],
      "source": [
        "# Convert structured output from response to data frame for display and/or further analysis\n",
        "video_extraction_response_df = pd.DataFrame(video_extraction_response.parsed)\n",
        "\n",
        "show(video_extraction_response_df)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfa2e8496790"
      },
      "source": [
        "## Creating insights from analyzing multiple YouTube videos together"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c634255fd419"
      },
      "source": [
        "### Google \"Year in Search\" videos\n",
        "Now, consider expanding the problem to a more common enterprise use case: extracting information from _multiple_ YouTube videos at once.\n",
        "\n",
        "This time, we'll use [Google's \"Year in Search\" videos](https://about.google/intl/ALL_us/stories/year-in-search/), which summarize the questions, people, and moments that captured the world's attention in each year. As of fall 2024, there are 14 of these videos, each 2-4 minutes in length, from [2010](https://www.youtube.com/watch?v=F0QXB5pw2qE) through [2023](https://www.youtube.com/watch?v=3KtWfp0UopM).\n",
        "\n",
        "We start by reading in a CSV file that has links to all the videos."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b004061c908a"
      },
      "outputs": [],
      "source": [
        "# Read in table of Year in Search video links from public CSV file\n",
        "GOOGLE_YEAR_IN_SEARCH_VIDEO_LINKS_CSV_GCS_URI = (\n",
        "    \"gs://github-repo/video/google_year_in_search_video_links.csv\"\n",
        ")\n",
        "\n",
        "year_in_search_yt_links = pd.read_csv(GOOGLE_YEAR_IN_SEARCH_VIDEO_LINKS_CSV_GCS_URI)\n",
        "\n",
        "year_in_search_yt_links"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "145522e33a47"
      },
      "source": [
        "### Set up for analyzing multiple video files\n",
        "\n",
        "Let's say we are a sports agency who wants to see which athletes or teams appear most often in these videos as a measure of cultural relevance. Instead of watching and manually counting, we can use Gemini's multimodal capabilities and world knowledge to extract each appearance of an athlete or team into a structured output that we can use for further analysis.\n",
        "\n",
        "The system instructions, prompt, and response schema that will apply to all 14 videos are each created in the cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "b8589a51547d"
      },
      "outputs": [],
      "source": [
        "# Set up pieces (prompt, response schema, config) for Google Year in Search videos\n",
        "multiple_video_extraction_system_instruction_text = (\n",
        "    \"You are a video analyst that \"\n",
        "    \"carefully looks through all frames of provided videos, extracting out the \"\n",
        "    \"pieces necessary to respond to user prompts.\"\n",
        ")\n",
        "\n",
        "multiple_video_extraction_prompt = (\n",
        "    \"Which sports athletes or teams are mentioned or \"\n",
        "    \"shown in this video? Please look through each frame carefully, and respond \"\n",
        "    \"with a complete list that includes the athlete or team's name (1 row per \"\n",
        "    \"athlete or team), whether they are an athlete or team, the sport they play, \"\n",
        "    \"and the timestamp into the video at which they appear (in mm:ss format, \"\n",
        "    \"do not give extra precision) for each one.\"\n",
        ")\n",
        "\n",
        "multiple_video_extraction_response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"OBJECT\",\n",
        "        \"properties\": {\n",
        "            \"name\": {\"type\": \"STRING\"},\n",
        "            \"athlete_or_team\": {\"type\": \"STRING\", \"enum\": [\"athlete\", \"team\"]},\n",
        "            \"sport\": {\"type\": \"STRING\"},\n",
        "            \"video_timestamp\": {\"type\": \"STRING\"},\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "multiple_video_extraction_json_generation_config = GenerateContentConfig(\n",
        "    temperature=0.0,\n",
        "    max_output_tokens=8192,\n",
        "    response_mime_type=\"application/json\",\n",
        "    response_schema=multiple_video_extraction_response_schema,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0cb2d4688f68"
      },
      "source": [
        "Next, we'll set up to run each of these prompt/video pairs through the Gemini API _asynchronously_. This allows us to send all the requests to Gemini at once, then wait for all the answers to come back - a more efficient process than sending them synchronously (one-by-one). See more details in [this Google Cloud Community Medium blog post](https://medium.com/google-cloud/how-to-prompt-gemini-asynchronously-using-python-on-google-cloud-986ca45d9f1b).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "5aa93ca907bc"
      },
      "outputs": [],
      "source": [
        "# Function for asynchronous generation\n",
        "\n",
        "\n",
        "@retry(wait=wait_random_exponential(multiplier=1, max=120), stop=stop_after_attempt(2))\n",
        "async def async_generate(prompt, yt_link):\n",
        "    try:\n",
        "        response = await client.aio.models.generate_content(\n",
        "            model=GEMINI_PRO_MODEL_ID,\n",
        "            contents=[prompt, Part.from_uri(file_uri=yt_link, mime_type=\"video/webm\")],\n",
        "            config=multiple_video_extraction_json_generation_config,\n",
        "        )\n",
        "\n",
        "        return response.to_json_dict()\n",
        "    except Exception as e:\n",
        "        print(\"Something failed, retrying\")\n",
        "        print(e)\n",
        "        with retry.stop_after_attempt(2) as retry_state:\n",
        "            if retry_state.attempt > 2:\n",
        "                return None\n",
        "        raise  # Re-raise the exception for tenacity to handle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61265bdff388"
      },
      "source": [
        "### Run asynchronous Gemini calls to do video extraction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4191dc30d77a"
      },
      "outputs": [],
      "source": [
        "# Perform asynchronous calls across all videos, gather responses\n",
        "import asyncio\n",
        "\n",
        "start_time = asyncio.get_event_loop().time()\n",
        "\n",
        "get_responses = [\n",
        "    async_generate(multiple_video_extraction_prompt, yt_link)\n",
        "    for yt_link in year_in_search_yt_links[\"yt_link\"]\n",
        "]\n",
        "\n",
        "multiple_video_extraction_responses = await asyncio.gather(*get_responses)\n",
        "\n",
        "end_time = asyncio.get_event_loop().time()\n",
        "\n",
        "elapsed_time = end_time - start_time\n",
        "\n",
        "print(f\"Elapsed time: {elapsed_time:.2f} seconds\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7c69057ae51d"
      },
      "source": [
        "### Extract and analyze video results across years\n",
        "\n",
        "Once we have the results from Gemini, we can process them and get table of every athlete or team appearance across all 14 \"Year in Search\" videos."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6e424adf2cf8"
      },
      "outputs": [],
      "source": [
        "# Add structured outputs by year back to original table, show full extraction results\n",
        "year_in_search_responses = year_in_search_yt_links.copy()\n",
        "\n",
        "year_in_search_responses[\"gemini_response\"] = [\n",
        "    json.dumps(response) for response in multiple_video_extraction_responses\n",
        "]\n",
        "\n",
        "\n",
        "def extract_result_df_from_gemini_response(year, gemini_response):\n",
        "    extract_response_text = json.loads(gemini_response)[\"candidates\"][0][\"content\"][\n",
        "        \"parts\"\n",
        "    ][0][\"text\"]\n",
        "\n",
        "    extract_result_df = pd.DataFrame(json.loads(extract_response_text))\n",
        "\n",
        "    extract_result_df[\"year\"] = year\n",
        "\n",
        "    return extract_result_df\n",
        "\n",
        "\n",
        "year_in_search_responses[\"extract_result_df\"] = year_in_search_responses.apply(\n",
        "    lambda row: extract_result_df_from_gemini_response(\n",
        "        row[\"year\"], row[\"gemini_response\"]\n",
        "    ),\n",
        "    axis=1,\n",
        ")\n",
        "\n",
        "all_year_in_search_extractions = pd.concat(\n",
        "    year_in_search_responses[\"extract_result_df\"].tolist(), ignore_index=True\n",
        ")[[\"year\", \"name\", \"athlete_or_team\", \"sport\", \"video_timestamp\"]]\n",
        "\n",
        "show(all_year_in_search_extractions)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b17e9b0af4e4"
      },
      "source": [
        "Finally, we can count the number of years in which each athlete or team appeared in these videos, and return results for those who appeared more than once."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c0cd6041bce7"
      },
      "outputs": [],
      "source": [
        "# Analyze results to show athletes/teams showing up most often in Year in Search videos\n",
        "multiple_year_in_search_app = (\n",
        "    all_year_in_search_extractions.assign(\n",
        "        # Convert 'name' to uppercase to handle e.g. \"LeBron\" vs \"Lebron\"\n",
        "        name=all_year_in_search_extractions[\"name\"].str.upper(),\n",
        "        # Convert 'athlete_or_team' to lowercase for consistency\n",
        "        athlete_or_team=all_year_in_search_extractions[\"athlete_or_team\"].str.lower(),\n",
        "    )\n",
        "    .groupby([\"name\", \"athlete_or_team\"])\n",
        "    .apply(\n",
        "        lambda x: pd.Series(\n",
        "            {\n",
        "                # Aggregate 'sport' across type and name (handling different cases)\n",
        "                \"sport\": \", \".join(sorted(x[\"sport\"].str.lower().unique())),\n",
        "                # Count # of diff years in which each athlete/team appears in video\n",
        "                \"num_years\": x[\"year\"].nunique(),\n",
        "            }\n",
        "        )\n",
        "    )\n",
        "    .reset_index()\n",
        "    .\n",
        "    # Filter to only those appearing multiple times\n",
        "    query(\"num_years >= 2\")\n",
        "    .sort_values([\"num_years\", \"name\"], ascending=[False, True])\n",
        "    .reset_index(drop=True)\n",
        ")\n",
        "\n",
        "# Display results\n",
        "display(Markdown(\"<b>Athletes/Teams Appearing in Multiple Year in Search Videos<b>\"))\n",
        "display(multiple_year_in_search_app)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "youtube_video_analysis.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
