{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yFeds7eiwI6x"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ircZQgHwRdy"
      },
      "source": [
        "# Intro to Computer Use with Gemini\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fcomputer-use%2Fintro_computer_use.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/computer-use/intro_computer_use.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<p>\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/computer-use/intro_computer_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>\n",
        "</p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MJDxNAGItD51"
      },
      "source": [
        "| Authors |\n",
        "| --- |\n",
        "| [Eric Dong](https://github.com/gericdong) |\n",
        "| [Holt Skinner](https://github.com/holtskinner) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "896Uhs2Ww6_E"
      },
      "source": [
        "## Overview\n",
        "\n",
        "The **Computer Use** model lets you create agents that can automate tasks on a computer. It works by \"seeing\" the screen with screenshots and then \"acting\" with mouse clicks and keyboard inputs.\n",
        "\n",
        "This is useful for tasks like:\n",
        "\n",
        "- Automatically filling out forms on websites.\n",
        "- Testing web applications.\n",
        "- Researching information, like comparing prices, across different sites.\n",
        "\n",
        "Learn more about [computer use model and tool](https://cloud.google.com/vertex-ai/generative-ai/docs/computer-use).\n",
        "\n",
        "\n",
        "## Objective\n",
        "\n",
        "In this tutorial, you will build a simple web automation agent using the Gemini Computer Use model. By the end, you will understand the complete workflow: from sending an initial prompt with a screenshot to executing browser actions and looping until a task is complete."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gPiTOAHURvTM"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CHRZUpfWSEpp"
      },
      "source": [
        "### Install the Gen AI SDK and required libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sG3_LKsWSD3A"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-genai playwright"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "msiBxrk0ATzB"
      },
      "source": [
        "> ⚠️ Note: You can ignore the pip's dependency errors."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WiZkhIF41qhY"
      },
      "source": [
        "### Set up Playwright\n",
        "\n",
        "Playwright is a tool for browser automation. It enables browser control over web browsers like Chromium, Firefox, and WebKit.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zL7fxJtq1cWT"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "\n",
        "# Installs Playwright and browsers\n",
        "!playwright install\n",
        "\n",
        "# Additional command, mandatory for Linux only\n",
        "!playwright install-deps"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HlMVjiAWSMNX"
      },
      "source": [
        "### Authenticate your notebook environment\n",
        "\n",
        "If you are running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "12fnq4V0SNV3"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rK3jJDR5lfiT"
      },
      "source": [
        "### Import libraries\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T6DyDoNclVEn"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import time\n",
        "from types import SimpleNamespace\n",
        "from typing import List, Tuple\n",
        "\n",
        "from google import genai\n",
        "from google.genai.types import (\n",
        "    ComputerUse,\n",
        "    Content,\n",
        "    Environment,\n",
        "    FunctionCall,\n",
        "    FunctionResponse,\n",
        "    FunctionResponseBlob,\n",
        "    GenerateContentConfig,\n",
        "    Part,\n",
        "    Tool,\n",
        ")\n",
        "from playwright.async_api import async_playwright"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "be18ac9c5ec8"
      },
      "source": [
        "### Set your project information\n",
        "\n",
        "Update the following variables with your Google Cloud project details, and connect to the Gen AI service on Vertex AI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6wXh1aH7BlPl"
      },
      "outputs": [],
      "source": [
        "# fmt: off\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "LOCATION = \"global\"  # @param {type: \"string\"}\n",
        "# fmt: on\n",
        "\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "# Connect to the Gen AI service on Vertex AI\n",
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n4yRkFg6BBu4"
      },
      "source": [
        "### Supported Models\n",
        "\n",
        "This tutorial uses the `gemini-2.5-computer-use-preview-10-2025` model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-coEslfWPrxo"
      },
      "outputs": [],
      "source": [
        "# fmt: off\n",
        "MODEL_ID = \"gemini-2.5-computer-use-preview-10-2025\"  # @param {type: \"string\"}\n",
        "# fmt: on"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KqjCltg20IgR"
      },
      "source": [
        "## Computer Use: Agent Loop\n",
        "\n",
        "To build a browser control agent, you implement an \"agent loop\" that continuously cycles through four key steps. This process allows the agent to perform a sequence of actions to achieve a goal.\n",
        "\n",
        "1. **Send Request to the Model**. Your app sends the goal (e.g., \"Find me a flight\") and a current screenshot of the screen to the model.\n",
        "\n",
        "2. **Receive the Model Response**. The model analyzes the screen and sends back a suggested action, like navigate to a URL. It may also include a safety warning for risky actions.\n",
        "\n",
        "3. **Execute the Received Action**. Your code runs the suggested action. If there's a safety warning, your code must ask the user for confirmation before proceeding.\n",
        "\n",
        "\n",
        "4. **Capture the New Environment State**. After the action, your code takes a new screenshot. This new screenshot is sent back to the model in the next turn, starting the cycle over again."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L85sHbJv9cpF"
      },
      "source": [
        "## Prerequisites: Setting Up Your Environment\n",
        "\n",
        "Before you begin, you need to set up two key components:\n",
        "\n",
        "- **Secure Execution Environment**: For safety, you must run your Computer Use agent in a secure and controlled environment. Good options include a sandboxed virtual machine, a container, or a dedicated browser profile with limited permissions.\n",
        "\n",
        "- **Client-Side Action Handler**: You need to write client-side logic to execute the actions generated by the model (e.g., clicking a button) and capture screenshots.\n",
        "\n",
        "In this tutorial, we use Playwright to start a browser environment for demonstration purpose."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vU5s-EBxCGiN"
      },
      "outputs": [],
      "source": [
        "# Start the Playwright session\n",
        "playwright = await async_playwright().start()\n",
        "\n",
        "# Launch the browser in headless mode, which is required for this environment\n",
        "browser = await playwright.chromium.launch(headless=True)\n",
        "\n",
        "# Create a new page\n",
        "page = await browser.new_page()\n",
        "screen_width, screen_height = 1920, 1080\n",
        "await page.set_viewport_size({\"width\": screen_width, \"height\": screen_height})\n",
        "\n",
        "print(\"Playwright session started.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nNhzdiH3-str"
      },
      "source": [
        "## A Single Turn: Step-by-Step Walkthrough\n",
        "\n",
        "Now, let's walk through the code for a single turn of the agent loop, from sending the first request to preparing for the next one.\n",
        "\n",
        "### **1. Send a Request to the Model**\n",
        "First, you configure your API request. In the request, you add the Computer Use tool and send a prompt that includes the user's goal and an initial screenshot.\n",
        "\n",
        "You can also include optional parameters like `excluded_predefined_functions` to prevent the model from using certain actions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M8CdwkdPAC8e"
      },
      "outputs": [],
      "source": [
        "# Configure Computer Use tool with browser environment\n",
        "config = GenerateContentConfig(\n",
        "    tools=[\n",
        "        Tool(\n",
        "            computer_use=ComputerUse(\n",
        "                environment=Environment.ENVIRONMENT_BROWSER,\n",
        "                # Optional: Exclude specific predefined functions\n",
        "                excluded_predefined_functions=[\"drag_and_drop\"],\n",
        "            )\n",
        "        )\n",
        "    ],\n",
        ")\n",
        "\n",
        "# Create the content with user message and initial screenshot\n",
        "screenshot_bytes = await page.screenshot()\n",
        "\n",
        "contents = [\n",
        "    Content(\n",
        "        role=\"user\",\n",
        "        parts=[\n",
        "            Part(\n",
        "                text=\"Find me a flight from SF to Hawaii on next Monday, coming back on next Friday. Start by navigating directly to flights.google.com\"\n",
        "            ),\n",
        "            # Optional: include a screenshot of the initial state\n",
        "            Part.from_bytes(\n",
        "                data=screenshot_bytes,\n",
        "                mime_type=\"image/png\",\n",
        "            ),\n",
        "        ],\n",
        "    )\n",
        "]\n",
        "\n",
        "# Generate content with the configured settings\n",
        "response = client.models.generate_content(\n",
        "    model=MODEL_ID,\n",
        "    contents=contents,\n",
        "    config=config,\n",
        ")\n",
        "\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M8ymc2GH_xxs"
      },
      "source": [
        "### **2. Receive the Model Response**\n",
        "The model responds with one or more `FunctionCalls` that represent the UI actions it wants to perform. Let's inspect the response from our first API call."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aC4wcsEBJ9vt"
      },
      "outputs": [],
      "source": [
        "response.function_calls"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vj9YwsxKvdg"
      },
      "source": [
        "### **3. Execute the Received Actions**\n",
        "\n",
        "Next, our application's client-side code needs to parse the response and execute the requested actions using Playwright. We'll use the `execute_function_calls` helper function for this.\n",
        "\n",
        "The following example implements some most common UI actions. For a production use case, you would need to implement all supported actions unless you explicitly exclude them."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tQqZqdTCf1YH"
      },
      "outputs": [],
      "source": [
        "def normalize_x(x: int, screen_width: int) -> int:\n",
        "    \"\"\"Convert normalized x coordinate (0-1000) to actual pixel coordinate.\"\"\"\n",
        "    return int(x / 1000 * screen_width)\n",
        "\n",
        "\n",
        "def normalize_y(y: int, screen_height: int) -> int:\n",
        "    \"\"\"Convert normalized y coordinate (0-1000) to actual pixel coordinate.\"\"\"\n",
        "    return int(y / 1000 * screen_height)\n",
        "\n",
        "\n",
        "async def execute_function_calls(\n",
        "    response, page, screen_width, screen_height\n",
        ") -> List[Tuple[str, str]]:\n",
        "    \"\"\"\n",
        "    Extracts and executes function calls from the model response.\n",
        "    \"\"\"\n",
        "    candidate = response.candidates[0]\n",
        "    function_calls = []\n",
        "    thoughts = []\n",
        "\n",
        "    for part in candidate.content.parts:\n",
        "        if hasattr(part, \"function_call\") and part.function_call:\n",
        "            function_calls.append(part.function_call)\n",
        "        elif hasattr(part, \"text\") and part.text:\n",
        "            thoughts.append(part.text)\n",
        "\n",
        "    if thoughts:\n",
        "        print(f\" Model Reasoning: {' '.join(thoughts)}\")\n",
        "\n",
        "    if not function_calls:\n",
        "        return \"NO_ACTION\", []\n",
        "\n",
        "    results = []\n",
        "    for function_call in function_calls:\n",
        "        result = None\n",
        "        print(f\"⚡ Executing Action: {function_call.name}\")\n",
        "        try:\n",
        "            if function_call.name == \"open_web_browser\":\n",
        "                result = \"success\"\n",
        "            elif function_call.name == \"navigate\":\n",
        "                await page.goto(function_call.args[\"url\"])\n",
        "                result = \"success\"\n",
        "            elif function_call.name == \"click_at\":\n",
        "                actual_x = normalize_x(function_call.args[\"x\"], screen_width)\n",
        "                actual_y = normalize_y(function_call.args[\"y\"], screen_height)\n",
        "                await page.mouse.click(actual_x, actual_y)\n",
        "                result = \"success\"\n",
        "            elif function_call.name == \"type_text_at\":\n",
        "                actual_x = normalize_x(function_call.args[\"x\"], screen_width)\n",
        "                actual_y = normalize_y(function_call.args[\"y\"], screen_height)\n",
        "                await page.mouse.click(actual_x, actual_y)\n",
        "                time.sleep(0.1)\n",
        "                await page.keyboard.type(function_call.args[\"text\"])\n",
        "                if function_call.args.get(\"press_enter\", False):\n",
        "                    await page.keyboard.press(\"Enter\")\n",
        "                result = \"success\"\n",
        "            else:\n",
        "                result = \"unknown_function\"\n",
        "        except Exception as e:\n",
        "            print(f\"❗️ Error executing {function_call.name}: {e}\")\n",
        "            result = f\"error: {str(e)}\"\n",
        "        results.append((function_call.name, result))\n",
        "    return \"CONTINUE\", results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_SP2x-IjMSzr"
      },
      "outputs": [],
      "source": [
        "results = await execute_function_calls(response, page, screen_width, screen_height)\n",
        "print(results)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y287MbtSJ7bs"
      },
      "source": [
        "Here is an example action for navigating a URL. In this case, we create a simple mock response object.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AubneGmKA2bw"
      },
      "outputs": [],
      "source": [
        "mock_response = SimpleNamespace(\n",
        "    candidates=[\n",
        "        SimpleNamespace(\n",
        "            content=SimpleNamespace(\n",
        "                parts=[\n",
        "                    SimpleNamespace(\n",
        "                        function_call=FunctionCall(\n",
        "                            name=\"navigate\", args={\"url\": \"https://flights.google.com\"}\n",
        "                        ),\n",
        "                    )\n",
        "                ]\n",
        "            )\n",
        "        )\n",
        "    ]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OW9N3x6kC-kc"
      },
      "outputs": [],
      "source": [
        "print(f\"Current page URL: {page.url}\")\n",
        "\n",
        "print(\"Calling execute_function_calls with a sample response\")\n",
        "results = await execute_function_calls(mock_response, page, screen_width, screen_height)\n",
        "print(f\"Results from execution:\\n{results}\\n\")\n",
        "\n",
        "print(f\"Navigated to: {page.url}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dkndDAPXNLQx"
      },
      "source": [
        "### **4. Capture the New State and Respond**\n",
        "\n",
        "Finally, after executing the actions, we capture a new screenshot and the current URL. This state information is then formatted as a `FunctionResponse` and added to our conversation history, making it ready for the next turn in the loop."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8QcMbBE5NUHU"
      },
      "outputs": [],
      "source": [
        "status, action_results_list = results\n",
        "\n",
        "function_response_parts = []\n",
        "\n",
        "for name, result in action_results_list:\n",
        "    # After each action, capture a new screenshot and the current URL\n",
        "    screenshot_bytes = await page.screenshot()\n",
        "    current_url = page.url\n",
        "\n",
        "    # Create a FunctionResponse for each action that was executed\n",
        "    # This is required even if multiple actions were called in parallel\n",
        "    function_response_parts.append(\n",
        "        FunctionResponse(\n",
        "            name=name,\n",
        "            response={\"url\": current_url},\n",
        "            parts=[\n",
        "                Part(\n",
        "                    inline_data=FunctionResponseBlob(\n",
        "                        mime_type=\"image/png\", data=screenshot_bytes\n",
        "                    )\n",
        "                )\n",
        "            ],\n",
        "        )\n",
        "    )\n",
        "\n",
        "# Package all the function responses into a single 'user' message\n",
        "user_feedback_content = Content(role=\"user\", parts=function_response_parts)\n",
        "\n",
        "# Append this new message to your conversation history\n",
        "contents.append(user_feedback_content)\n",
        "\n",
        "print(\"Step 4 Complete: New state captured and added to conversation history.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YAZBbH3hD1y8"
      },
      "source": [
        "The `contents` list is now ready for the next call to the model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O3LJGCkHD640"
      },
      "source": [
        "#### Clean up by closing the browser and stopping the session\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GKoItatcCqrV"
      },
      "outputs": [],
      "source": [
        "await browser.close()\n",
        "await playwright.stop()\n",
        "\n",
        "print(\"Browser closed and Playwright session stopped.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZR7UOfgOdji"
      },
      "source": [
        "## Build an Agent Loop\n",
        "\n",
        "To enable multi-step interactions, combine the four steps from the How to implement Computer Use section into a loop. The loop must handle parallel function calls, and safety decisions. Remember to manage the conversation history correctly by appending both model responses and your function responses."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "m5mYV8CbZE05"
      },
      "outputs": [],
      "source": [
        "async def agent_loop(initial_prompt, max_turns=10):\n",
        "    \"\"\"Main agent loop\"\"\"\n",
        "    playwright_loop = await async_playwright().start()\n",
        "    browser_loop = await playwright_loop.chromium.launch(headless=True)\n",
        "    page_loop = await browser_loop.new_page()\n",
        "    sw, sh = 1920, 1080\n",
        "    await page_loop.set_viewport_size({\"width\": sw, \"height\": sh})\n",
        "\n",
        "    print(f\"Starting Agent Loop with prompt: '{initial_prompt}'\")\n",
        "\n",
        "    screenshot = await page_loop.screenshot()\n",
        "    contents = [\n",
        "        Content(\n",
        "            role=\"user\",\n",
        "            parts=[\n",
        "                Part(text=initial_prompt),\n",
        "                Part.from_bytes(data=screenshot, mime_type=\"image/png\"),\n",
        "            ],\n",
        "        )\n",
        "    ]\n",
        "\n",
        "    for turn in range(max_turns):\n",
        "        print(f\"\\n Turn {turn + 1}\")\n",
        "\n",
        "        response = client.models.generate_content(\n",
        "            model=MODEL_ID, contents=contents, config=config\n",
        "        )\n",
        "\n",
        "        # Handle cases where the model returns no candidates (e.g., due to safety filters)\n",
        "        if not response.candidates:\n",
        "            print(\"Model returned no candidates. This may be due to a safety filter.\")\n",
        "            print(\"Full Response:\", response)\n",
        "            print(\"Terminating loop.\")\n",
        "            break\n",
        "\n",
        "        contents.append(response.candidates[0].content)\n",
        "\n",
        "        function_calls = [\n",
        "            part.function_call\n",
        "            for part in response.candidates[0].content.parts\n",
        "            if hasattr(part, \"function_call\") and part.function_call\n",
        "        ]\n",
        "\n",
        "        # Finish the agent loop if no function call in the response.\n",
        "        if not function_calls:\n",
        "            final_text = \"\".join(\n",
        "                part.text\n",
        "                for part in response.candidates[0].content.parts\n",
        "                if hasattr(part, \"text\") and part.text is not None\n",
        "            )\n",
        "            if final_text:\n",
        "                print(f\"Agent Finished: {final_text}\")\n",
        "                break\n",
        "\n",
        "        status, execution_results = await execute_function_calls(\n",
        "            response, page_loop, sw, sh\n",
        "        )\n",
        "\n",
        "        if status == \"NO_ACTION\":\n",
        "            continue\n",
        "\n",
        "        function_response_parts = []\n",
        "        for name, result in execution_results:\n",
        "            screenshot = await page_loop.screenshot()\n",
        "            current_url = page_loop.url\n",
        "            function_response_parts.append(\n",
        "                FunctionResponse(\n",
        "                    name=name,\n",
        "                    response={\"url\": current_url},\n",
        "                    parts=[\n",
        "                        Part(\n",
        "                            inline_data=FunctionResponseBlob(\n",
        "                                mime_type=\"image/png\", data=screenshot\n",
        "                            )\n",
        "                        )\n",
        "                    ],\n",
        "                )\n",
        "            )\n",
        "        contents.append(Content(role=\"user\", parts=function_response_parts))\n",
        "        print(f\"State captured. History now has {len(contents)} messages.\")\n",
        "\n",
        "    print(\"\\n Agent loop finished. Closing browser.\")\n",
        "    await browser_loop.close()\n",
        "    await playwright_loop.stop()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hX30dnPKnXYK"
      },
      "outputs": [],
      "source": [
        "# RUN THE AGENT LOOP\n",
        "prompt = \"Navigate to the Google Store and find the 'Pixel' category.\"\n",
        "\n",
        "await agent_loop(prompt)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "intro_computer_use.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
