{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EWeQ3WOjAzDa"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lvYXxb3XCf0W"
      },
      "source": [
        "# Gemini Live API Quickstart\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fmultimodal-live-api%2Flive_api_quickstart.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<p>\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/live_api_quickstart.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "76b2vG9eCxLW"
      },
      "source": [
        "| Authors |\n",
        "| --- |\n",
        "| [Eric Dong](https://github.com/gericdong) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfARntTuC2wn"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demonstrates how to connect to the **Gemini Live API** for real-time, bidirectional audio streaming. You will learn to establish a session with a Gemini model, simulate an audio stream from a file, and play back the generated audio response.\n",
        "\n",
        "The guide provides two implementation examples:\n",
        "\n",
        "- [Gen AI SDK](https://github.com/googleapis/python-genai): A simplified approach using the Google **Gen AI SDK** to manage the session and handle interruptions.\n",
        "\n",
        "- [WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API): A low-level approach using standard **WebSockets** to construct the handshake and manage raw JSON payloads."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kQX8-lbsC8JJ"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_0OdbM-RDDw2"
      },
      "source": [
        "### Install required libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0JwX5N8fDhnc"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade google-genai"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EVo5SimcDWKF"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BRtdID2HsInB"
      },
      "outputs": [],
      "source": [
        "import asyncio\n",
        "import base64\n",
        "import os\n",
        "import sys\n",
        "import json\n",
        "import wave\n",
        "import websockets\n",
        "import numpy as np\n",
        "\n",
        "from IPython.display import Audio, display\n",
        "\n",
        "from google import genai\n",
        "from google.genai import types"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WBZPC10PDcxk"
      },
      "source": [
        "### Authenticate your notebook environment\n",
        "\n",
        "If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V-Ry6-JdDYKz"
      },
      "outputs": [],
      "source": [
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mdTSBOzZDhie"
      },
      "source": [
        "### Authenticate your Google Cloud Project for Vertex AI\n",
        "\n",
        "You can use a Google Cloud Project or an API Key for authentication. This tutorial uses a Google Cloud Project.\n",
        "\n",
        "- [Enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AR6vYdRTsSfv"
      },
      "outputs": [],
      "source": [
        "# fmt: off\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "# fmt: on\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = \"us-central1\"  # @param {type: \"string\", placeholder: \"global\"}\n",
        "\n",
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFozk2URDrTE"
      },
      "source": [
        "### Choose a Gemini model\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "lBEljCMawnyN"
      },
      "outputs": [],
      "source": [
        "# fmt: off\n",
        "MODEL_ID = \"gemini-live-2.5-flash-native-audio\"  # @param {type: \"string\"}\n",
        "# fmt: on"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z5mKXikOEVjC"
      },
      "source": [
        "### About audio streaming\n",
        "\n",
        "In these guides, you send an audio file to the model and receive audio in response. In production, the input audio would be a microphone stream. The Live API supports the following audio formats:\n",
        "\n",
        "- **Input audio**: Raw 16-bit PCM audio at 16kHz, little-endian\n",
        "- **Output audio**: Raw 16-bit PCM audio at 24kHz, little-endian\n",
        "\n",
        "The client must maintain a playback buffer. The server streams audio in chunks (within server_content messages). The client's responsibility is to:\n",
        "\n",
        "- **Decode**: Base64 decode the `inline_data`.\n",
        "- **Buffer**: Append the binary data to a queue.\n",
        "- **Play**: Feed the data to the audio hardware.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bwUgw2r6u7Sh"
      },
      "outputs": [],
      "source": [
        "# Download a sample audio input file\n",
        "audio_file = \"input.wav\"\n",
        "audio_file_url = f\"https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/tell-a-story.wav\"\n",
        "\n",
        "!wget -q $audio_file_url -O $audio_file\n",
        "\n",
        "with wave.open(audio_file, 'rb') as wf:\n",
        "    frames = wf.readframes(wf.getnframes())\n",
        "    print(f\"Read audio: {len(frames)} bytes\")\n",
        "    print(f\"Channels: {wf.getnchannels()}\")\n",
        "    print(f\"Rate: { wf.getframerate()}Hz\")\n",
        "    print(f\"Width: {wf.getsampwidth()} bytes\")\n",
        "\n",
        "display(Audio(filename=audio_file, autoplay=True))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5FLPsMtxFZcD"
      },
      "source": [
        "### 🚀 Quickstart 1: Using Gen AI SDK\n",
        "\n",
        "In this quickstart, you learn how to connect to the Live API using the [Google Gen AI SDK](https://github.com/googleapis/python-genai), and establish a session, send an audio file to the model and receive audio in response. This is a simplified approach to manage the session and handle interruptions.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hsE6yc94sn6i"
      },
      "outputs": [],
      "source": [
        "# Configuration\n",
        "config = {\n",
        "    \"response_modalities\": [\"audio\"],\n",
        "}\n",
        "\n",
        "\n",
        "async def main():\n",
        "    # Establish WebSocket session\n",
        "    async with client.aio.live.connect(model=MODEL_ID, config=config) as session:\n",
        "        print(\"Session established. Sending audio...\")\n",
        "\n",
        "        # Send Input (Simulated from file)\n",
        "        # In production, this would be a microphone stream\n",
        "        with open(\"input.wav\", \"rb\") as f:\n",
        "            while chunk := f.read(1024):\n",
        "                await session.send_realtime_input(\n",
        "                    audio=types.Blob(data=chunk, mime_type=\"audio/pcm;rate=16000\")\n",
        "                )\n",
        "                await asyncio.sleep(0.01)  # Simulate real-time stream\n",
        "\n",
        "        audio_data = []\n",
        "\n",
        "        # Receive Output\n",
        "        async for message in session.receive():\n",
        "            if message.server_content:\n",
        "                # Check for interruptions (User barge-in)\n",
        "                if message.server_content.interrupted:\n",
        "                    print(\"[Interrupted] Clear client audio buffer immediately.\")\n",
        "                    continue\n",
        "\n",
        "                # Process Audio Chunks\n",
        "                model_turn = message.server_content.model_turn\n",
        "                if model_turn and model_turn.parts:\n",
        "                    for part in model_turn.parts:\n",
        "                        if part.inline_data:\n",
        "                            # Output is PCM, 24kHz, 16-bit, Mono\n",
        "                            print(\n",
        "                                f\"Received audio chunk: {len(part.inline_data.data)} bytes\"\n",
        "                            )\n",
        "                            audio_data.append(\n",
        "                                np.frombuffer(part.inline_data.data, dtype=np.int16)\n",
        "                            )\n",
        "\n",
        "            if message.server_content.turn_complete:\n",
        "                print(\"Turn complete.\")\n",
        "                display(Audio(np.concatenate(audio_data), rate=24000, autoplay=True))\n",
        "\n",
        "\n",
        "# Run directly in notebook\n",
        "await main()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjtDVNQNG6eD"
      },
      "source": [
        "## 🚀 Quickstart 2: Using WebSocket\n",
        "\n",
        "In this quickstart, you learn how to connect to the Live API using [WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API), and send an audio file to the model and receive audio in response. This is a low-level approach using standard WebSockets to construct the handshake and manage raw JSON payloads."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Id-qQxr2ytE3"
      },
      "outputs": [],
      "source": [
        "# Authentication\n",
        "token_list = !gcloud auth application-default print-access-token\n",
        "headers = {\"Authorization\": f\"Bearer {token_list[0]}\"}\n",
        "\n",
        "# Configuration\n",
        "MODEL = f\"projects/{PROJECT_ID}/locations/{LOCATION}/publishers/google/models/{MODEL_ID}\"\n",
        "config = {\n",
        "   \"response_modalities\": [\"audio\"],\n",
        "}\n",
        "\n",
        "# Construct the WSS URL\n",
        "HOST = f\"{LOCATION}-aiplatform.googleapis.com\"\n",
        "URI = f\"wss://{HOST}/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent\"\n",
        "\n",
        "async def main():\n",
        "    # Connect to the server\n",
        "    async with websockets.connect(URI, additional_headers=headers) as ws:\n",
        "      # Send Setup (Handshake)\n",
        "      await ws.send(json.dumps({\n",
        "          \"setup\": {\n",
        "              \"model\": MODEL,\n",
        "              \"generation_config\": config\n",
        "          }\n",
        "      }))\n",
        "      print(\"Session established. Sending audio...\")\n",
        "\n",
        "      # Receive setup response\n",
        "      raw_response = await ws.recv(decode=False)\n",
        "      setup_response = json.loads(raw_response.decode(\"ascii\"))\n",
        "\n",
        "      # Send Input (Simulated from file)\n",
        "      # In production, this would be a microphone stream\n",
        "      with open(\"input.wav\", \"rb\") as f:\n",
        "          while chunk := f.read(1024):\n",
        "              msg = {\n",
        "                  \"realtime_input\": {\n",
        "                      \"media_chunks\": [{\n",
        "                          \"mime_type\": \"audio/pcm;rate=16000\",\n",
        "                          \"data\": base64.b64encode(chunk).decode(\"utf-8\")\n",
        "                      }]\n",
        "                  }\n",
        "              }\n",
        "              await ws.send(json.dumps(msg))\n",
        "              await asyncio.sleep(0.01) # Simulate real-time stream\n",
        "\n",
        "      audio_data = []\n",
        "\n",
        "      # Receive chunks of server response\n",
        "      async for raw_response in ws:\n",
        "          data = json.loads(raw_response.decode())\n",
        "          try:\n",
        "              parts = data[\"serverContent\"][\"modelTurn\"][\"parts\"]\n",
        "              for part in parts:\n",
        "                  if \"inlineData\" in part:\n",
        "                      b64_audio = part[\"inlineData\"][\"data\"]\n",
        "                      print(f\"Received chunk: {len(b64_audio)} bytes\")\n",
        "                      pcm_data = base64.b64decode(b64_audio)\n",
        "                      audio_data.append(np.frombuffer(pcm_data, dtype=np.int16))\n",
        "          except KeyError:\n",
        "              pass\n",
        "\n",
        "          if data.get(\"serverContent\", {}).get(\"turnComplete\"):\n",
        "              print(\"Turn complete.\")\n",
        "              display(Audio(np.concatenate(audio_data), rate=24000, autoplay=True))\n",
        "              break\n",
        "\n",
        "\n",
        "# Run directly in notebook\n",
        "await main()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3qNOSDcLBMq"
      },
      "source": [
        "## What's next\n",
        "\n",
        "Now that you have established a basic connection to the Gemini Live API, try these advanced capabilities to build a production-ready application:\n",
        "\n",
        "- [Getting Started with the Live API in Vertex AI using WebSockets](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api.ipynb)\n",
        "- [Getting Started with the Live API using Gen AI SDK](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/multimodal-live-api/intro_multimodal_live_api_genai_sdk.ipynb)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fZAJ1pkNw26D"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
