{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OsXAs2gcIpbC"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WKiAD3qCCtOu"
      },
      "source": [
        " # Evaluate Generative Model Tool Use"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaltask_approach%2Fevaluate_gemini_tool_use.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/evaluate_gemini_tool_use.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0922hY8RI11o"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Jason Dai](https://github.com/jsondai),  [Bo Zheng](https://github.com/coolalexzb) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmNXwtYGCtOv"
      },
      "source": [
        "## Overview\n",
        "\n",
        "* Define an API function and a Tool for Gemini model, and evaluate the Gemini model tool use quality with *Vertex AI Python SDK for Gen AI Evaluation Service*.\n",
        "\n",
        "See also: \n",
        "\n",
        "- Learn more about [Vertex Gen AI Evaluation Service SDK](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58zvbIBKCtOv"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHKucTJ9CtOv"
      },
      "source": [
        "### Install Vertex AI Python SDK for Gen AI Evaluation Service"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6DnneQRilTFW"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nWC8oDQCCtOw"
      },
      "source": [
        "### Restart runtime\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
        "\n",
        "The restart might take a minute or longer. After it's restarted, continue to the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dCkOvWuOCtOw"
      },
      "outputs": [],
      "source": [
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h4pNZoFdCtOw"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NdCE9EV5CtOw"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZiX59CNjCtOx"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4FOrb_sCtOx"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "G2LhRUspCtOx"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    raise ValueError(\"Please set your PROJECT_ID\")\n",
        "\n",
        "\n",
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1EvqYQT2CtOx"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eHDPrnbQlTFX"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "import pandas as pd\n",
        "from vertexai.evaluation import EvalTask\n",
        "from vertexai.generative_models import GenerativeModel\n",
        "from vertexai.preview.evaluation import notebook_utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RPBjzk96w9gV"
      },
      "source": [
        "## Evaluate Tool use and Function Calling quality for Gemini"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iY5hG6VMnS_s"
      },
      "source": [
        "#### Tool evaluation metrics\n",
        "\n",
        "* `tool_call_valid`\n",
        "* `tool_name_match`\n",
        "* `tool_parameter_key_match`\n",
        "* `tool_parameter_kv_match`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vcZ2UikInZe0"
      },
      "outputs": [],
      "source": [
        "tool_metrics = [\n",
        "    \"tool_call_valid\",\n",
        "    \"tool_name_match\",\n",
        "    \"tool_parameter_key_match\",\n",
        "    \"tool_parameter_kv_match\",\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dJgpakfIw9gc"
      },
      "source": [
        "### 1. Evaluate a Bring-Your-Own-Prediction dataset\n",
        "\n",
        "Generative model's tool use quality can be evaluated if the eval dataset contains saved model tool call responses, and expected references."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d6Euv-SQw9gc"
      },
      "outputs": [],
      "source": [
        "response = [\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\"}}]}',\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Cinemark\", \"location\": \"Mountain View CA\", \"showtime\": \"5:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
        "]\n",
        "\n",
        "reference = [\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Godzilla\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"9:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
        "    '{\"content\": \"\", \"tool_calls\": [{\"name\": \"book_tickets\", \"arguments\": {\"movie\": \"Mission Impossible Dead Reckoning Part 1\", \"theater\": \"Regal Edwards 14\", \"location\": \"Mountain View CA\", \"showtime\": \"7:30\", \"date\": \"2024-03-30\", \"num_tix\": \"2\"}}]}',\n",
        "]\n",
        "\n",
        "eval_dataset = pd.DataFrame(\n",
        "    {\n",
        "        \"response\": response,\n",
        "        \"reference\": reference,\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PyUj7kOnnNtH"
      },
      "source": [
        "#### Define EvalTask"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tYl6A1zInObM"
      },
      "outputs": [],
      "source": [
        "experiment_name = \"eval-saved-llm-tool-use\"  # @param {type:\"string\"}\n",
        "\n",
        "tool_use_eval_task = EvalTask(\n",
        "    dataset=eval_dataset,\n",
        "    metrics=tool_metrics,\n",
        "    experiment=experiment_name,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "l8U4KVzAhsP2"
      },
      "outputs": [],
      "source": [
        "run_id = notebook_utils.generate_uuid(8)\n",
        "\n",
        "experiment_run_name = f\"eval-{run_id}\"\n",
        "\n",
        "eval_result = tool_use_eval_task.evaluate(experiment_run_name=experiment_run_name)\n",
        "notebook_utils.display_eval_result(\n",
        "    title=\"Tool Use Quality Evaluation Metrics\",\n",
        "    eval_result=eval_result,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3WXnsT2qo-OD"
      },
      "outputs": [],
      "source": [
        "tool_use_eval_task.display_runs()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxow7GrIRgxo"
      },
      "source": [
        "## 2. Tool Use and Function Calling with Gemini\n",
        "\n",
        "[Function Calling Documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ew82QX0QWxYa"
      },
      "source": [
        "### Define a function and tool\n",
        "\n",
        "Define an API specification and register the function in a tool with the latest version of [Vertex AI SDK for Python](https://cloud.google.com/vertex-ai/docs/python-sdk/use-vertex-ai-python-sdk).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LiuafKieQgjF"
      },
      "outputs": [],
      "source": [
        "from vertexai.generative_models import FunctionDeclaration, Tool\n",
        "\n",
        "book_tickets_func = FunctionDeclaration(\n",
        "    name=\"book_tickets\",\n",
        "    description=\"Book movie tickets\",\n",
        "    parameters={\n",
        "        \"type\": \"object\",\n",
        "        \"properties\": {\n",
        "            \"movie\": {\"type\": \"string\", \"description\": \"The title of the movie.\"},\n",
        "            \"theater\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The name of the movie theater.\",\n",
        "            },\n",
        "            \"location\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The location of the movie theater.\",\n",
        "            },\n",
        "            \"showtime\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The showtime of the movie in ISO 8601 format.\",\n",
        "            },\n",
        "            \"date\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The date of the movie in ISO 8601 format.\",\n",
        "            },\n",
        "            \"num_tix\": {\n",
        "                \"type\": \"string\",\n",
        "                \"description\": \"The integer number of tickets to book.\",\n",
        "            },\n",
        "        },\n",
        "        \"required\": [\n",
        "            \"movie\",\n",
        "            \"theater\",\n",
        "            \"location\",\n",
        "            \"showtime\",\n",
        "            \"date\",\n",
        "            \"num_tix\",\n",
        "        ],\n",
        "    },\n",
        ")\n",
        "\n",
        "\n",
        "book_tickets_tool = Tool(\n",
        "    function_declarations=[book_tickets_func],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQgreOFGXJNF"
      },
      "source": [
        "### Generate a function call\n",
        "\n",
        "Prompt the Gemini model and include the tool that you defined."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "pNevmS2ngB0Z"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"I'd like to book 2 tickets for the movie \"Mission Impossible Dead Reckoning Part 1\"\n",
        "at the Regal Edwards 14 theater in Mountain View, CA. The showtime is 7:30 PM on March 30th, 2024.\n",
        "\"\"\"\n",
        "\n",
        "gemini_model = GenerativeModel(\"gemini-2.0-flash\")\n",
        "\n",
        "gemini_response = gemini_model.generate_content(\n",
        "    prompt,\n",
        "    tools=[book_tickets_tool],\n",
        ")\n",
        "\n",
        "gemini_response.candidates[0].content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lOAbbxFFXkdn"
      },
      "source": [
        "###  Unpack the Gemini response into a Python dictionary"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x84XHpuWXmu6"
      },
      "outputs": [],
      "source": [
        "def unpack_response(response):\n",
        "    output = {}\n",
        "    function_call = {}\n",
        "    for key, value in response.candidates[0].content.parts[0].to_dict().items():\n",
        "        function_call[key] = value\n",
        "    output[\"content\"] = \"\"\n",
        "    output[\"tool_calls\"] = [function_call[\"function_call\"]]\n",
        "    output[\"tool_calls\"][0][\"arguments\"] = output[\"tool_calls\"][0].pop(\"args\")\n",
        "    return json.dumps(output)\n",
        "\n",
        "\n",
        "response = unpack_response(gemini_response)\n",
        "response"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fsLT0B75aJYz"
      },
      "source": [
        "### Evaluate the Gemini's Function Call Response"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kgzK90mybqbK"
      },
      "outputs": [],
      "source": [
        "reference_str = json.dumps(\n",
        "    {\n",
        "        \"content\": \"\",\n",
        "        \"tool_calls\": [\n",
        "            {\n",
        "                \"name\": \"book_tickets\",\n",
        "                \"arguments\": {\n",
        "                    \"movie\": \"Mission Impossible Dead Reckoning Part 1\",\n",
        "                    \"theater\": \"Regal Edwards 14\",\n",
        "                    \"location\": \"Mountain View CA\",\n",
        "                    \"showtime\": \"7:30\",\n",
        "                    \"date\": \"2024-03-30\",\n",
        "                    \"num_tix\": \"2\",\n",
        "                },\n",
        "            }\n",
        "        ],\n",
        "    }\n",
        ")\n",
        "\n",
        "eval_dataset = pd.DataFrame({\"response\": [response], \"reference\": [reference_str]})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ttRycKaCg6GG"
      },
      "outputs": [],
      "source": [
        "# Expected Tool Call Response\n",
        "json.loads(eval_dataset.reference[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k5SmLtFAdVCh"
      },
      "outputs": [],
      "source": [
        "# Actual Gemini Tool Call Response\n",
        "json.loads(eval_dataset.response[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r3MfX9QCZxHW"
      },
      "outputs": [],
      "source": [
        "experiment_name = \"eval-gemini-model-function-call\"  # @param {type:\"string\"}\n",
        "run_id = notebook_utils.generate_uuid(8)\n",
        "\n",
        "eval_result = EvalTask(\n",
        "    dataset=eval_dataset,\n",
        "    metrics=tool_metrics,\n",
        "    experiment=experiment_name,\n",
        ").evaluate(experiment_run_name=f\"eval-{run_id}\")\n",
        "\n",
        "notebook_utils.display_eval_result(\n",
        "    title=\"Gemini Tool Use Quality Evaluation Metrics\",\n",
        "    eval_result=eval_result,\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "evaluate_gemini_tool_use.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
