{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "77d580c8",
      "metadata": {
        "id": "77d580c8"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d2a6d1bc",
      "metadata": {
        "id": "d2a6d1bc"
      },
      "source": [
        "# AI Quick Build Experience Backend\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fuse-cases%2Feducation%2Fai_quick_build_experience_backend.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/use-cases/education/ai_quick_build_experience_backend.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2f4b6b36",
      "metadata": {
        "id": "2f4b6b36"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Zack Akil](https://github.com/zackail)|"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e65a3863",
      "metadata": {
        "id": "e65a3863"
      },
      "source": [
        "## Overview\n",
        "\n",
        "The AI Quick Build experience is a Gemini and Imagen powered game show experience where contestants race against the clock to construct the best interpretation of a given object, e.g., a \"lighthouse.\" Gemini provides live commentary and performs the final judging of whose build is the best. Then, Imagen takes what has been built and brings it to life.\n",
        "\n",
        " This notebook provides an insight into how the AI Quick Build experience was built and how you can build similar features into your applications.\n",
        "\n",
        "The notebook is divided into three main sections:\n",
        "\n",
        "1. **Real-time commentary:** This section demonstrates how to generate live commentary using Gemini Flash and the Cloud Text-to-Speech API.\n",
        "2. **Judging:** This section showcases how to use Gemini for in-depth analysis and judgment of the builds, providing detailed reasoning and a winner announcement script.\n",
        "3. **Image generation:** This section explores how to enhance the winner's creation by generating a dynamic background using Imagen, and then creating an animated GIF that transitions between the original and generated images.\n",
        "\n",
        "### Objective\n",
        "\n",
        "This notebook aims to:\n",
        "\n",
        "* Provide a clear understanding of how the AI Quick Build experience was developed.\n",
        "* Enable users to build similar experiences by demonstrating the capabilities of Gemini and Imagen.\n",
        "* Inspire users to explore new applications of these technologies in areas like education and entertainment.\n",
        "\n",
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "- Vertex AI\n",
        "- Cloud Text-to-Speech\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Text-to-Speech pricing](https://cloud.google.com/text-to-speech/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "951930c4",
      "metadata": {
        "id": "951930c4"
      },
      "source": [
        "## Get Started"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e31deb12",
      "metadata": {
        "id": "e31deb12"
      },
      "source": [
        "### Install Vertex AI SDK and other required packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "428311af",
      "metadata": {
        "id": "428311af"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-cloud-texttospeech \\\n",
        "                        google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9005c308",
      "metadata": {
        "id": "9005c308"
      },
      "source": [
        "### Restart runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
        "\n",
        "The restart might take a minute or longer. After it's restarted, continue to the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "49b60539",
      "metadata": {
        "id": "49b60539"
      },
      "outputs": [],
      "source": [
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b78c686d",
      "metadata": {
        "id": "b78c686d"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9f0de68e",
      "metadata": {
        "id": "9f0de68e"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "06a9699c",
      "metadata": {
        "id": "06a9699c"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"YOUR PROJECT ID\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user(project_id=PROJECT_ID)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "51ad2c80",
      "metadata": {
        "id": "51ad2c80"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "80af9f03",
      "metadata": {
        "id": "80af9f03"
      },
      "outputs": [],
      "source": [
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "81a6e2f3",
      "metadata": {
        "id": "81a6e2f3"
      },
      "source": [
        "# Part 1: Real-Time Commentary\n",
        "\n",
        "This section demonstrates how to use Gemini Flash and the Cloud Text-to-Speech API to generate real-time commentary on the builds as they are being constructed.\n",
        "\n",
        "Gemini Flash's low latency allows us to provide immediate feedback and analysis, enhancing the interactive nature of the experience.\n",
        "\n",
        "We will also showcase the power of using Google Cloud's new Casual voices for text-to-speech, which can create a more natural and engaging commentary experience."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "87798176",
      "metadata": {
        "id": "87798176"
      },
      "source": [
        "### Setting up the Commentator's Voice\n",
        "\n",
        "Here we define a helper function that utilizes the Cloud Text-to-Speech API to generate speech from a given text script. We will use this function to give our commentator a voice."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cc85c36f",
      "metadata": {
        "id": "cc85c36f"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "import IPython\n",
        "from IPython import display\n",
        "from google.cloud import texttospeech\n",
        "\n",
        "texttospeech_client = texttospeech.TextToSpeechClient()\n",
        "\n",
        "\n",
        "def speak(script: str, voice_name: str = \"en-US-Casual-K\") -> IPython.lib.display.Audio:\n",
        "    \"\"\"\n",
        "    Generates speech from the given script using the specified voice and plays it.\n",
        "\n",
        "    Args:\n",
        "        script: The text script to convert to speech.\n",
        "        voice_name: The name of the voice to use (default: 'en-US-Casual-K').\n",
        "    \"\"\"\n",
        "    generate_speech(script, voice_name)\n",
        "    return display.Audio(\"output.wav\")\n",
        "\n",
        "\n",
        "def generate_speech(\n",
        "    script: str,\n",
        "    voice_name: str = \"en-US-Casual-K\",\n",
        "    output_file_name: str = \"output.wav\",\n",
        ") -> str:\n",
        "    \"\"\"Generates a wav audio file from the given script using Google Cloud Text-to-Speech.\n",
        "\n",
        "    Args:\n",
        "        script: The text script to convert to speech.\n",
        "        voice_name: The name of the voice to use (e.g., 'en-US-Casual-K').\n",
        "        output_file_name: The name of the output MP3 file.\n",
        "    \"\"\"\n",
        "    synthesis_input = texttospeech.SynthesisInput(text=script)\n",
        "\n",
        "    language_code = extract_language_code(voice_name)\n",
        "\n",
        "    voice = texttospeech.VoiceSelectionParams(\n",
        "        language_code=language_code, name=voice_name\n",
        "    )\n",
        "\n",
        "    # Select the type of audio file you want returned\n",
        "    audio_config = texttospeech.AudioConfig(\n",
        "        audio_encoding=texttospeech.AudioEncoding.LINEAR16\n",
        "    )\n",
        "\n",
        "    # Perform the text-to-speech request on the text input with the selected\n",
        "    tts_response = texttospeech_client.synthesize_speech(\n",
        "        input=synthesis_input, voice=voice, audio_config=audio_config\n",
        "    )\n",
        "\n",
        "    # The response's audio_content is binary.\n",
        "    with open(output_file_name, \"wb\") as out:\n",
        "        # Write the response to the output file.\n",
        "        out.write(tts_response.audio_content)\n",
        "        print(f\"Audio content written to file {output_file_name}\")\n",
        "    return output_file_name\n",
        "\n",
        "\n",
        "def extract_language_code(voice_name: str) -> str | None:\n",
        "    \"\"\"Extracts the language code by splitting on the second hyphen.\"\"\"\n",
        "    parts = voice_name.split(\"-\", 2)  # Split into max 3 parts\n",
        "    if len(parts) >= 2:\n",
        "        return \"-\".join(parts[:2])  # Join the first two parts\n",
        "    else:\n",
        "        return None\n",
        "\n",
        "\n",
        "# Demonstrate the casual voice\n",
        "speak(\"Hello, I'm ready to speak\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c692e79c",
      "metadata": {
        "id": "c692e79c"
      },
      "source": [
        "### Generating Live Commentary with Gemini Flash and Cloud Text-to-Speech\n",
        "\n",
        "In this section, we'll use Gemini Flash to generate commentary based on images of the players' builds. We'll then use the `speak` function we defined earlier to convert this commentary into speech."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "oLEJ918-y6hE",
      "metadata": {
        "id": "oLEJ918-y6hE"
      },
      "outputs": [],
      "source": [
        "# Import vertex ai classes\n",
        "from vertexai.generative_models import GenerationConfig, GenerativeModel, Image, Part"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "d002764c",
      "metadata": {
        "id": "d002764c"
      },
      "outputs": [],
      "source": [
        "# Fetch images of players' builds from Google Cloud Storage\n",
        "!gcloud storage cp gs://cloud-samples-data/generative-ai/image/quick_build_* ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "5827c565",
      "metadata": {
        "id": "5827c565"
      },
      "outputs": [],
      "source": [
        "image_urls = [\n",
        "    \"quick_build_1.png\",\n",
        "    \"quick_build_2.png\",\n",
        "    \"quick_build_3.png\",\n",
        "    \"quick_build_4.png\",\n",
        "]\n",
        "\n",
        "# Load the images using the Vertex AI SDK Image loader\n",
        "all_images = [Image.load_from_file(url) for url in image_urls]\n",
        "\n",
        "# We could also load the images directly from Google Cloud Storage with Image.from_uri(url)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f0fbf92f",
      "metadata": {
        "id": "f0fbf92f"
      },
      "outputs": [],
      "source": [
        "# Define the player names\n",
        "player_names = [\"Elara\", \"Finn\", \"Juniper\", \"Rhys\"]\n",
        "\n",
        "# Display the player build images\n",
        "for i, img in enumerate(all_images):\n",
        "    print(f\"Player {i+1}, {player_names[i]}:\")\n",
        "    display.display(img)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "4d17e406",
      "metadata": {
        "id": "4d17e406"
      },
      "outputs": [],
      "source": [
        "# Define the design challenge\n",
        "design = \"lighthouse\"\n",
        "\n",
        "# Initialize a list to store previous comments\n",
        "previous_comments = []\n",
        "\n",
        "# Define the commentator's persona\n",
        "commentator_persona = \"\"\"\n",
        "whimsical, but can be quippy\n",
        "\"\"\"\n",
        "\n",
        "# Define the prompt for generating commentary\n",
        "commentary_prompt = f\"\"\"\n",
        "You are a commentator at a high-stakes toy brick building contest where participants\n",
        "have only 100 seconds to construct a {design}.\n",
        "They will be judged on the following Creative\n",
        "Use of Color\n",
        "Structural Integrity:\n",
        "Adherence to Brief:\n",
        "Detail and Complexity:\n",
        "Make a short comment about one of the players' builds so far.\n",
        "Make it specific about something visual.\n",
        "Make sure to call out the player by name.\n",
        "\n",
        "Your persona is {commentator_persona}.\n",
        "\n",
        "Here are the previous comments said:\n",
        "--- Start of previous comments ---\n",
        "{previous_comments}\n",
        "--- End of previous comments ---\n",
        "\n",
        "Mix up who the comments are about so that everyone is included,\n",
        "but only talk about the players mentioned below:\n",
        "Here are the latest photos of all of the builds:\n",
        "\"\"\"\n",
        "\n",
        "# Initialize the prompt parts with the base prompt\n",
        "prompt_parts = [commentary_prompt]\n",
        "\n",
        "# Add player information and images to the prompt parts\n",
        "for i, name in enumerate(player_names):\n",
        "    prompt_player_part = f\"Player {i+1}, {name}:\"\n",
        "    prompt_parts.append(prompt_player_part)\n",
        "\n",
        "    players_photo = all_images[i]\n",
        "    prompt_photo_part = Part.from_image(players_photo)\n",
        "\n",
        "    prompt_parts.append(prompt_photo_part)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9d5105e1",
      "metadata": {
        "id": "9d5105e1"
      },
      "source": [
        "### Leveraging Controlled Generation to Structure the Output\n",
        "\n",
        "Here, we will use **Controlled Generation** to ensure that Gemini Flash outputs the commentary in a specific JSON format. This will allow us to easily parse the response and extract the information we need, such as the commentary script, the player the commentary is about, and their number.\n",
        "\n",
        "Learn more about Controlled Generation: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "777f2d82",
      "metadata": {
        "id": "777f2d82"
      },
      "outputs": [],
      "source": [
        "# Load the Gemini Flash model\n",
        "gemini_flash = GenerativeModel(\"gemini-2.0-flash\")\n",
        "\n",
        "# Define the response schema for controlled generation\n",
        "response_schema = {\n",
        "    \"type\": \"object\",\n",
        "    \"properties\": {\n",
        "        \"commentaryScript\": {\"type\": \"string\"},\n",
        "        \"playerNameCommentaryIsAbout\": {\"type\": \"string\"},\n",
        "        \"playerNumberCommentaryIsAbout\": {\"type\": \"integer\"},\n",
        "    },\n",
        "    \"required\": [\n",
        "        \"commentaryScript\",\n",
        "        \"playerNameCommentaryIsAbout\",\n",
        "        \"playerNumberCommentaryIsAbout\",\n",
        "    ],\n",
        "}\n",
        "\n",
        "# Configure the generation settings, including the response MIME type and schema\n",
        "generation_config = GenerationConfig(\n",
        "    response_mime_type=\"application/json\",\n",
        "    response_schema=response_schema,\n",
        "    temperature=0.7,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "a8953fa9",
      "metadata": {
        "id": "a8953fa9"
      },
      "outputs": [],
      "source": [
        "# Generate commentary using Gemini Flash and the defined prompt parts and configuration\n",
        "commentary = gemini_flash.generate_content(\n",
        "    prompt_parts, generation_config=generation_config\n",
        ")\n",
        "\n",
        "# Parse the JSON response\n",
        "commentary_dict = json.loads(commentary.text)\n",
        "commentary_script = commentary_dict.get(\"commentaryScript\")\n",
        "player_name_commentary_is_about = commentary_dict.get(\"playerNameCommentaryIsAbout\")\n",
        "player_number_commentary_is_about = commentary_dict.get(\"playerNumberCommentaryIsAbout\")\n",
        "\n",
        "# Print the extracted information\n",
        "print(commentary_script)\n",
        "print(player_name_commentary_is_about)\n",
        "print(player_number_commentary_is_about)\n",
        "\n",
        "# Display the image of the player being commented on\n",
        "all_images[player_number_commentary_is_about - 1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "42a7fb17",
      "metadata": {
        "id": "42a7fb17"
      },
      "outputs": [],
      "source": [
        "# Convert the commentary script to speech and play it\n",
        "speak(commentary_script)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "72130a81",
      "metadata": {
        "id": "72130a81"
      },
      "source": [
        "# Part 2: Contest Judging with Gemini\n",
        "\n",
        "This section demonstrates how to use Gemini to judge the completed builds. Gemini offers more complex and in-depth reasoning capabilities, making it ideal for evaluating the builds based on specific criteria and selecting an overall winner."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "4364dbff",
      "metadata": {
        "id": "4364dbff"
      },
      "outputs": [],
      "source": [
        "# Define the design challenge\n",
        "design = \"lighthouse\"\n",
        "\n",
        "# Define the judging prompt\n",
        "judging_prompt = f\"\"\"You are judging a high-stakes toy brick building contest where\n",
        "participants have only 100 seconds to construct a {design}.\n",
        "\n",
        " Evaluate each entry on a scale of 1-10 based on 'Creativity' & 'Execution' the following criteria:\n",
        " Creative Use of Color: Assess how intentionally and effectively different colored bricks have been used\n",
        " to enhance the overall design and convey specific themes or emotions.\n",
        " Structural Integrity: Consider the stability and durability of the build.\n",
        "  Does the structure hold together under scrutiny, and does it demonstrate a sound understanding of LEGO mechanics?\n",
        "  Adherence to Brief: Evaluate how closely the design aligns with the given prompt or theme.\n",
        "  Does it capture the essence of the concept and avoid unnecessary deviations?\n",
        "  Detail and Complexity: Assess the level of detail and complexity incorporated into the build.\n",
        "  Do the smaller elements contribute to the overall aesthetic and functionality?\n",
        "\n",
        "  And choose an overall winner!\n",
        "  Output an \"announcement script\" that builds up to announcing the winner. Keep it concise\n",
        "  And include a \"winner reasoning\" that goes into specific details about why that winner was chosen. Keep it concise\n",
        "  Here are the finished builds:\"\n",
        "  \"\"\"\n",
        "\n",
        "# Initialize the prompt parts with the judging prompt\n",
        "judging_prompt_parts = [judging_prompt]\n",
        "\n",
        "# Define the player names\n",
        "player_names = [\"Elara\", \"Finn\", \"Juniper\", \"Rhys\"]\n",
        "\n",
        "# Add player information and images to the judging prompt parts\n",
        "for i, name in enumerate(player_names):\n",
        "    prompt_player_part = f\"Player {i+1}, {name}:\"\n",
        "    judging_prompt_parts.append(prompt_player_part)\n",
        "\n",
        "    players_photo = all_images[i]\n",
        "    prompt_photo_part = Part.from_image(players_photo)\n",
        "\n",
        "    judging_prompt_parts.append(prompt_photo_part)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "90f99ae0",
      "metadata": {
        "id": "90f99ae0"
      },
      "outputs": [],
      "source": [
        "# Define the base response schema for judging\n",
        "judging_response_schema_base = {\n",
        "    \"type\": \"object\",\n",
        "    \"properties\": {\n",
        "        \"scores\": {\n",
        "            \"type\": \"object\",\n",
        "            \"properties\": {},\n",
        "        },\n",
        "        \"overallWinner\": {\n",
        "            \"type\": \"object\",\n",
        "            \"properties\": {\n",
        "                \"playerName\": {\n",
        "                    \"type\": \"string\",\n",
        "                },\n",
        "                \"playerNumber\": {\n",
        "                    \"type\": \"number\",\n",
        "                },\n",
        "                \"winnerReasoning\": {\n",
        "                    \"type\": \"string\",\n",
        "                    \"description\": \"specific details about why this winner was chosen.\",\n",
        "                },\n",
        "            },\n",
        "        },\n",
        "        \"winnerAnnouncementScript\": {\n",
        "            \"type\": \"string\",\n",
        "            \"description\": \"This is what the announcer will read out verbatim, comment on some specific aspects of all of the contestants builds, keeping the winner ambiguous to the very last word, e.g '... and that is why the winner is... [insert player name]!'\",\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "# Define the response schema for individual players' scores\n",
        "judging_response_schema_player = {\n",
        "    \"type\": \"object\",\n",
        "    \"properties\": {\n",
        "        \"playerName\": {\n",
        "            \"type\": \"string\",\n",
        "        },\n",
        "        \"playerNumber\": {\n",
        "            \"type\": \"number\",\n",
        "        },\n",
        "        \"score\": {\n",
        "            \"type\": \"number\",\n",
        "            \"description\": \"score between 1-10 for their overall build\",\n",
        "        },\n",
        "        \"scoringReasoning\": {\n",
        "            \"type\": \"string\",\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "# Add player-specific schemas to the base schema dynamically\n",
        "for i, name in enumerate(player_names):\n",
        "    judging_response_schema_base[\"properties\"][\"scores\"][\"properties\"][\n",
        "        f\"player_{i+1}\"\n",
        "    ] = judging_response_schema_player"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "3dfb907b",
      "metadata": {
        "id": "3dfb907b"
      },
      "outputs": [],
      "source": [
        "# Load the Gemini model\n",
        "gemini_pro = GenerativeModel(\"gemini-2.0-flash\")\n",
        "\n",
        "# Configure the generation settings for judging\n",
        "generation_config = GenerationConfig(\n",
        "    response_mime_type=\"application/json\",\n",
        "    response_schema=judging_response_schema_base,\n",
        "    temperature=0.7,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "3a930b96",
      "metadata": {
        "id": "3a930b96"
      },
      "outputs": [],
      "source": [
        "# Generate judging results using Gemini and the defined prompt parts and configuration\n",
        "judging = gemini_pro.generate_content(\n",
        "    judging_prompt_parts, generation_config=generation_config\n",
        ")\n",
        "\n",
        "# Parse the JSON response\n",
        "judging_dict = json.loads(judging.text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ffac0121",
      "metadata": {
        "id": "ffac0121"
      },
      "outputs": [],
      "source": [
        "# Display the judging results\n",
        "judging_dict"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "db33a3f4",
      "metadata": {
        "id": "db33a3f4"
      },
      "outputs": [],
      "source": [
        "# Extract information about the winner and the announcement script\n",
        "winning_player_name = judging_dict[\"overallWinner\"][\"playerName\"]\n",
        "winning_player_number = judging_dict[\"overallWinner\"][\"playerNumber\"]\n",
        "winning_player_reasoning = judging_dict[\"overallWinner\"][\"winnerReasoning\"]\n",
        "winner_announcement_script = judging_dict[\"winnerAnnouncementScript\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "id": "b930d0c7",
      "metadata": {
        "id": "b930d0c7"
      },
      "outputs": [],
      "source": [
        "# Print and speak the winner announcement script\n",
        "print(winner_announcement_script)\n",
        "# speak(winner_announcement_script)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "7b0924f2",
      "metadata": {
        "id": "7b0924f2"
      },
      "outputs": [],
      "source": [
        "# Announce the winner and display their image\n",
        "print(f\"Winner is {winning_player_name}!\")\n",
        "all_images[winning_player_number - 1]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "id": "eeb118c3",
      "metadata": {
        "id": "eeb118c3"
      },
      "outputs": [],
      "source": [
        "# Print and speak the winner reasoning\n",
        "print(winning_player_reasoning)\n",
        "speak(winning_player_reasoning)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fccb7305",
      "metadata": {
        "id": "fccb7305"
      },
      "source": [
        "# Part 3: Bringing the Creation to Life with Imagen\n",
        "\n",
        "In this section, we will use Imagen to enhance the winning build by generating a dynamic and engaging background. We will then create a fun animated GIF that transitions between the original build image and the Imagen-generated image.\n",
        "\n",
        "This showcases how Imagen can be used to create compelling visuals that complement and enhance user-generated content."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "id": "8ff8abf2",
      "metadata": {
        "id": "8ff8abf2"
      },
      "outputs": [],
      "source": [
        "from vertexai.preview.vision_models import Image as Imagen_Image\n",
        "from vertexai.preview.vision_models import ImageGenerationModel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "id": "389668b0",
      "metadata": {
        "id": "389668b0"
      },
      "outputs": [],
      "source": [
        "# Load the Imagen model\n",
        "imagen_model = ImageGenerationModel.from_pretrained(\"imagegeneration@006\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f8358a05",
      "metadata": {
        "id": "f8358a05"
      },
      "outputs": [],
      "source": [
        "# Load the winning build's image\n",
        "selected_image = image_urls[winning_player_number - 1]\n",
        "\n",
        "base_img = Imagen_Image.load_from_file(location=selected_image)\n",
        "base_img.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "id": "21a1850a",
      "metadata": {
        "id": "21a1850a"
      },
      "outputs": [],
      "source": [
        "# Define the prompt for Imagen to generate a background\n",
        "image_prompt = \"a macro lens photo of a toy brick lighthouse in a rocky stormy costal toy brick sea scene. The sea stretches out into the distance\"\n",
        "\n",
        "# Generate images using Imagen's edit_image function with background masking\n",
        "images = imagen_model.edit_image(\n",
        "    prompt=image_prompt,\n",
        "    base_image=base_img,\n",
        "    mask_mode=\"background\",\n",
        "    mask_dilation=0.003,  # Use a small mask dilation to avoid erasing parts of the subject\n",
        "    number_of_images=3,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "78552a40",
      "metadata": {
        "id": "78552a40"
      },
      "outputs": [],
      "source": [
        "# Display the generated images\n",
        "for i, img in enumerate(images):\n",
        "    print(\"generated image \", i)\n",
        "    img.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "id": "94c7a4b7",
      "metadata": {
        "id": "94c7a4b7"
      },
      "outputs": [],
      "source": [
        "# Select a favourite generated image\n",
        "favourite_generated_image = 2\n",
        "\n",
        "# Save the favourite generated image\n",
        "images[favourite_generated_image].save(\"favourite_generated_image.png\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b0b5c963",
      "metadata": {
        "id": "b0b5c963"
      },
      "source": [
        "### Creating an Animated GIF\n",
        "\n",
        "Now, let's create a fun GIF that fades between the original build image and the Imagen-generated image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "id": "1e9f2bd2",
      "metadata": {
        "id": "1e9f2bd2"
      },
      "outputs": [],
      "source": [
        "# import Pillow package for gif building\n",
        "import PIL"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "id": "6065033a",
      "metadata": {
        "id": "6065033a"
      },
      "outputs": [],
      "source": [
        "# Load the start and end images for the GIF\n",
        "start_image_url = image_urls[winning_player_number - 1]\n",
        "end_image_url = \"favourite_generated_image.png\"\n",
        "\n",
        "start_image = PIL.Image.open(start_image_url).convert(\"RGB\")\n",
        "end_image = PIL.Image.open(end_image_url).convert(\"RGB\")\n",
        "\n",
        "# Initialize a list to store the GIF frames\n",
        "gif_frames = []\n",
        "\n",
        "# Define the number of steps for the fade transition\n",
        "gif_steps = 3\n",
        "\n",
        "# Generate the frames for the fade-in\n",
        "for i in range(gif_steps + 1):\n",
        "    gif_frame = PIL.Image.blend(start_image, end_image, i / gif_steps)\n",
        "    gif_frames.append(gif_frame)\n",
        "\n",
        "# Add the end frame again so that it pauses at the generated image.\n",
        "gif_frames.append(end_image)\n",
        "\n",
        "# Add the frames for the fade-out (reverse of fade-in)\n",
        "gif_frames += gif_frames[::-1]\n",
        "\n",
        "# Save the frames as a GIF\n",
        "start_image.save(\n",
        "    \"generated.gif\",\n",
        "    format=\"GIF\",\n",
        "    append_images=gif_frames,\n",
        "    save_all=True,\n",
        "    duration=300,\n",
        "    loop=0,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "id": "328c9afd",
      "metadata": {
        "id": "328c9afd"
      },
      "outputs": [],
      "source": [
        "# Display the GIF\n",
        "display.Image(filename=\"generated.gif\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "25983d78",
      "metadata": {
        "id": "25983d78"
      },
      "source": [
        "# Conclusion\n",
        "\n",
        "This notebook has demonstrated how to build the backend for a fun and interactive game show experience using Gemini and Imagen. We have explored various capabilities of these models, including:\n",
        "\n",
        "* **Real-time commentary generation with Gemini Flash:** Providing immediate feedback and analysis on player builds.\n",
        "* **In-depth judging and winner selection with Gemini:** Leveraging advanced reasoning capabilities for a more comprehensive evaluation.\n",
        "* **Enhanced visuals with Imagen:** Bringing the winning build to life with a dynamically generated background and an animated GIF.\n",
        "\n",
        "### Potential Applications\n",
        "\n",
        "The techniques and concepts presented in this notebook can be applied to various applications beyond game shows, such as:\n",
        "\n",
        "* **Education:** Creating interactive learning experiences with personalized feedback and engaging visuals.\n",
        "* **Content creation:** Generating dynamic content for social media, marketing campaigns, and more.\n",
        "* **Entertainment:** Building interactive narratives and immersive experiences.\n",
        "\n",
        "### Learn More\n",
        "\n",
        "To further explore the capabilities of Gemini, Imagen, and other Generative AI technologies, check out the following resources:\n",
        "\n",
        "* [Gemini documentation](https://cloud.google.com/vertex-ai)\n",
        "* [Imagen documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/image/overview)\n",
        "* [Vertex AI Generative AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/overview)\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "ai_quick_build_experience_backend.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
