{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KjTHAV8FgEza"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UdL4uvQQs76x"
      },
      "source": [
        "# Veo 3 Advanced Controls\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fveo3_advanced_controls.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/veo3_advanced_controls.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/veo3_advanced_controls.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lUCUMoTmN_lJ"
      },
      "source": [
        "| Author |\n",
        "| --- |\n",
        "| [Katie Nguyen](https://github.com/katiemn) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rDjAqcgigwdX"
      },
      "source": [
        "## Overview\n",
        "\n",
        "### Veo 3\n",
        "\n",
        "Veo 3 on Vertex AI gives application developers access to Google's cutting-edge video generation. This model creates videos with stunning detail and realistic physics across a wide array of visual styles. Veo 3 enhances video quality from text and image prompts, and now includes dialogue and audio generation.\n",
        "\n",
        "In this tutorial, you will learn how to use the Google Gen AI SDK for Python to interact with Veo 3.1 advanced control features, including:\n",
        "  - Frame interpolation\n",
        "  - Video extend\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v2_iOv5uhXVg"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K4uerc9Xhf1f"
      },
      "source": [
        "### Install Google Gen AI SDK for Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rJyFNKoQhiwF"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWYnCW0-h6HI"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the following cell to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bqz5LUG6h8fA"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LVrasKoriKZn"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c5XgNQ2HD_HH"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import time\n",
        "import urllib.request\n",
        "\n",
        "import matplotlib.image as img\n",
        "import matplotlib.pyplot as plt\n",
        "from IPython.display import Video, display\n",
        "from PIL import Image as PIL_Image\n",
        "from google import genai\n",
        "from google.genai import types"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yxBkUEqdiB1g"
      },
      "source": [
        "### Set Google Cloud project information and create client\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GtjPBmYHiEfx"
      },
      "outputs": [],
      "source": [
        "# fmt: off\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "# fmt: on\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
        "\n",
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qD_bwA9hiMzL"
      },
      "source": [
        "### Define a helper function to display media"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GUrEwbvFiPhJ"
      },
      "outputs": [],
      "source": [
        "def show_video(video: str | bytes) -> None:\n",
        "    if isinstance(video, str):\n",
        "        file_name = video.split(\"/\")[-1]\n",
        "        !gsutil cp {video} {file_name}\n",
        "        display(Video(file_name, embed=True, width=600))\n",
        "    else:\n",
        "        with open(\"sample.mp4\", \"wb\") as out_file:\n",
        "            out_file.write(video)\n",
        "        display(Video(\"sample.mp4\", embed=True, width=600))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2jaSOOadiUj6"
      },
      "source": [
        "### Load the video generation model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "APRfTklCiYR2"
      },
      "outputs": [],
      "source": [
        "video_model = \"veo-3.1-generate-001\"\n",
        "video_extension_model = \"veo-3.1-generate-preview\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_1-VPOexGWe8"
      },
      "source": [
        "## Video generation with Veo 3.1 advanced controls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P604bVwQeDPN"
      },
      "source": [
        "### Frame interpolation\n",
        "\n",
        "With frame interpolation, you can specify the first and last frame of the video clip, and Veo will generate the rest.\n",
        "\n",
        "#### Load the images\n",
        "\n",
        "Download the first and last frames that you will use in the request. If you would like to use local images, simply comment out the `wget` command and edit the `first_frame` and `last_frame` variables.\n",
        "\n",
        "**Note:** This feature tends to work best when the starting and ending frames are similar in nature."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5-fpFTzwefcH"
      },
      "outputs": [],
      "source": [
        "# Download first frame\n",
        "!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/cookies.png\n",
        "# fmt: off\n",
        "first_frame = \"cookies.png\"  # @param {type: 'string'}\n",
        "\n",
        "# Download last frame\n",
        "!wget https://storage.googleapis.com/cloud-samples-data/generative-ai/image/cookies-milk.png\n",
        "last_frame = \"cookies-milk.png\"  # @param {type: 'string'}\n",
        "# fmt: on\n",
        "\n",
        "# Display the images\n",
        "fig, axis = plt.subplots(1, 2, figsize=(12, 6))\n",
        "axis[0].imshow(img.imread(first_frame))\n",
        "axis[0].set_title(\"First Frame\")\n",
        "axis[1].imshow(img.imread(last_frame))\n",
        "axis[1].set_title(\"Last Frame\")\n",
        "for ax in axis:\n",
        "    ax.axis(\"off\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yKR0riV_K5nT"
      },
      "source": [
        "With Veo 3.1, you have the option to generate videos with audio from a text prompt and/or input image(s). In order to generate a video in the following sample, specify the following info:\n",
        "- **Prompt:** A description of the video you would like to see. Since you're starting with images you can also leave the prompt field empty.\n",
        "- **Aspect ratio:** Select either 16:9 or 9:16.\n",
        "- **Resolution:** 720p, 1080p or 4k.\n",
        "- **Number of videos:** Set this value to 1, 2, 3, or 4.\n",
        "- **Video duration:** Can be 4, 6, or 8 seconds.\n",
        "- **Person generation:** Set to `allow_adult` or `dont_allow`.\n",
        "- **Audio generation:** Set `generate_audio` to True if you'd like audio included in your generated video.\n",
        "\n",
        "\n",
        "**Safety:** All Veo videos include [SynthID](https://deepmind.google/technologies/synthid/), which embeds a digital watermark directly into the AI-generated video.\n",
        "\n",
        "**NOTE:** Generating a video in 4k will introduce increased latency up to several minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3YzkQI5JekEh"
      },
      "outputs": [],
      "source": [
        "# fmt: off\n",
        "prompt = \"a hand reaches in and places a glass of milk next to the plate of cookies\"  # @param {type: 'string'}\n",
        "# fmt: on\n",
        "\n",
        "operation = client.models.generate_videos(\n",
        "    model=video_model,\n",
        "    prompt=prompt,\n",
        "    image=types.Image.from_file(location=first_frame),\n",
        "    config=types.GenerateVideosConfig(\n",
        "        last_frame=types.Image.from_file(location=last_frame),\n",
        "        aspect_ratio=\"9:16\",\n",
        "        resolution=\"1080p\",\n",
        "        number_of_videos=1,\n",
        "        duration_seconds=8,\n",
        "        person_generation=\"allow_adult\",\n",
        "        generate_audio=True,\n",
        "    ),\n",
        ")\n",
        "\n",
        "while not operation.done:\n",
        "    time.sleep(10)\n",
        "    operation = client.operations.get(operation)\n",
        "    print(operation)\n",
        "\n",
        "if operation.response:\n",
        "    show_video(operation.result.generated_videos[0].video.video_bytes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rln2_Oi3Piu-"
      },
      "source": [
        "In this next example, you'll continue to use first and last frames stored in Cloud Storage. However, in this request, you'll reference the images in Cloud Storage directly, rather than downloading them locally first. If you'd like to use a different Cloud Storage image, replace the URL and `gcs_uri` variables below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aCzame3wGEpi"
      },
      "outputs": [],
      "source": [
        "first_frame = PIL_Image.open(\n",
        "    urllib.request.urlopen(\n",
        "        \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/forest-1.png\"\n",
        "    )\n",
        ")\n",
        "first_frame_gcs = \"gs://cloud-samples-data/generative-ai/image/forest-1.png\"\n",
        "\n",
        "last_frame = PIL_Image.open(\n",
        "    urllib.request.urlopen(\n",
        "        \"https://storage.googleapis.com/cloud-samples-data/generative-ai/image/forest-2.png\"\n",
        "    )\n",
        ")\n",
        "last_frame_gcs = \"gs://cloud-samples-data/generative-ai/image/forest-2.png\"\n",
        "\n",
        "# Display the images\n",
        "fig, axis = plt.subplots(1, 2, figsize=(18, 6))\n",
        "axis[0].imshow(first_frame)\n",
        "axis[1].imshow(last_frame)\n",
        "for ax in axis:\n",
        "    ax.axis(\"off\")\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XPu4QrPAGanp"
      },
      "source": [
        "Rather than output `video_bytes` in this section, you'll save your video to Cloud Storage. In order to accomplish this, set your Cloud Storage bucket location in `output_gcs`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ylUHNer1GleN"
      },
      "outputs": [],
      "source": [
        "# fmt: off\n",
        "prompt = \"the chipmunk climbs down the tree to sit next to the rabbit\"  # @param {type: 'string'}\n",
        "# fmt: on\n",
        "output_gcs = \"gs://[your-bucket-path]\"  # @param {type: 'string'}\n",
        "\n",
        "operation = client.models.generate_videos(\n",
        "    model=video_model,\n",
        "    prompt=prompt,\n",
        "    image=types.Image(gcs_uri=first_frame_gcs, mime_type=\"image/png\"),\n",
        "    config=types.GenerateVideosConfig(\n",
        "        last_frame=types.Image(gcs_uri=last_frame_gcs, mime_type=\"image/png\"),\n",
        "        output_gcs_uri=output_gcs,\n",
        "        aspect_ratio=\"16:9\",\n",
        "        number_of_videos=1,\n",
        "        duration_seconds=6,\n",
        "        resolution=\"1080p\",\n",
        "        person_generation=\"allow_adult\",\n",
        "        generate_audio=True,\n",
        "    ),\n",
        ")\n",
        "\n",
        "while not operation.done:\n",
        "    time.sleep(10)\n",
        "    operation = client.operations.get(operation)\n",
        "    print(operation)\n",
        "\n",
        "if operation.response:\n",
        "    show_video(operation.result.generated_videos[0].video.uri)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gg6mf3iWbjRP"
      },
      "source": [
        "### Video extend\n",
        "\n",
        "In this next example, you'll use Veo to extend an existing video. You'll use the same parameters as before. However, the following are configured differently in this instance:\n",
        "- **Input video:** Since you're starting with a video, specify a Cloud Storage video location in the `video_gcs` variable.\n",
        "- **Output video:** As these videos will be bigger, you'll save your output video in Cloud Storage. Include the Cloud Storage bucket in the `output_gcs` field.\n",
        "- **Video duration:** You can extend your video by 7 seconds."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-dXqvIuAbzVQ"
      },
      "outputs": [],
      "source": [
        "prompt = \"a butterfly flies in and lands on the flower\"  # @param {type: 'string'}\n",
        "# fmt: off\n",
        "video_gcs = \"gs://cloud-samples-data/generative-ai/video/red-flower.mp4\"  # @param {type: 'string'}\n",
        "# fmt: on\n",
        "output_gcs = \"gs://[your-bucket-path]\"  # @param {type: 'string'}\n",
        "\n",
        "operation = client.models.generate_videos(\n",
        "    model=video_extension_model,\n",
        "    prompt=prompt,\n",
        "    video=types.Video(uri=video_gcs, mime_type=\"video/mp4\"),\n",
        "    config=types.GenerateVideosConfig(\n",
        "        output_gcs_uri=output_gcs,\n",
        "        number_of_videos=1,\n",
        "        duration_seconds=7,\n",
        "        person_generation=\"allow_adult\",\n",
        "        generate_audio=True,\n",
        "    ),\n",
        ")\n",
        "\n",
        "while not operation.done:\n",
        "    time.sleep(5)\n",
        "    operation = client.operations.get(operation)\n",
        "    print(operation)\n",
        "\n",
        "if operation.response:\n",
        "    show_video(operation.result.generated_videos[0].video.uri)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "veo3_advanced_controls.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
