{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Evaluate Gemini Structured Output\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main//gemini/evaluation/evaluate_gemini_structured_output.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaluate_gemini_structured_output.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main//gemini/evaluation/evaluate_gemini_structured_output.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaluate_gemini_structured_output.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84f0f73a0f76"
      },
      "source": [
        "| Author |\n",
        "| --- |\n",
        "| [Steve Phillips](https://github.com/stevie-p) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook uses the [**Gen AI Evaluation Service**](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation) to evaluate and compare the performance of Gemini models for an extraction task.\n",
        "\n",
        "The task is to accurately extract information from a scanned, handwritten order form for \"Acme Corporation\".\n",
        "\n",
        "Within this notebook, we:\n",
        "\n",
        "* Use Gemini models with [structured output](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output) to ensure well-structured JSON output\n",
        "* Extract the data using Gemini models\n",
        "* Define two custom metrics: `valid_schema` using the [`jsonschema`](https://pypi.org/project/jsonschema/) library, and `accuracy` using the [`deepdiff`](https://github.com/seperman/deepdiff) library\n",
        "* Use the **Gen AI Evaluation service** to run the evaluation experiments\n",
        "\n",
        "The [models](https://cloud.google.com/vertex-ai/generative-ai/docs/models) under test are:\n",
        "\n",
        "* Gemini 2.0 Flash\n",
        "* Gemini 2.5 Flash\n",
        "* Gemini 2.5 Pro"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Google Gen AI SDK and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-genai jsonschema IPython==7.34.0 google-cloud-aiplatform 'pybind11>=2.12' deepdiff"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "795uBvHqqy4V"
      },
      "source": [
        "Restart the runtime to use the newly installed packages."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "# Create your own project and insert the project ID here ---->\n",
        "\n",
        "import os\n",
        "\n",
        "# fmt: off\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "# fmt: on\n",
        "LOCATION = \"us-central1\"  # @param {type: \"string\"}\n",
        "\n",
        "# Use the environment variable if the user doesn't provide Project ID.\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "if not LOCATION:\n",
        "    LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
        "\n",
        "EXPERIMENT_NAME = \"eval-gemini-structured\"\n",
        "\n",
        "import vertexai\n",
        "from google import genai\n",
        "\n",
        "client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5303c05f7aa6"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6fc324893334"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "import pandas as pd\n",
        "from IPython.display import Image, Markdown, display\n",
        "from deepdiff import DeepDiff\n",
        "from google.genai.types import (\n",
        "    GenerateContentConfig,\n",
        "    Part,\n",
        ")\n",
        "from jsonschema import validate\n",
        "from vertexai.evaluation import CustomMetric, EvalTask, notebook_utils"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9Q3T_1ox99_"
      },
      "source": [
        "## View the images\n",
        "\n",
        "Let's have a look at the images we want to extract data from."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q7u9dNhytp5s"
      },
      "outputs": [],
      "source": [
        "images = [\n",
        "    {\n",
        "        \"image_url\": \"https://storage.googleapis.com/github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/AcmeOrderForm.jpg\",\n",
        "        \"image_uri\": \"gs://github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/AcmeOrderForm.jpg\",\n",
        "        \"image_type\": \"image/jpeg\",\n",
        "        \"image_name\": \"Acme Order Form.jpg\",\n",
        "        \"reference\": {  # The Ground Truth\n",
        "            \"order_number\": \"98-X42-77A\",\n",
        "            \"order_date\": \"2025-09-01\",\n",
        "            \"customer_name\": \"WILE E. COYOTE (ESQ., PH.D, S.G.)\",\n",
        "            \"customer_address\": \"HIGH MESA, CORNER OF X-MARK AND DETONATION CANYON, ANVIL FALLS, AZ\",\n",
        "            \"line_items\": [\n",
        "                {\n",
        "                    \"item_description\": \"Jet Propelled Unicycle\",\n",
        "                    \"quantity\": 1,\n",
        "                    \"unit_price\": 99.99,\n",
        "                    \"delivery_option\": \"Next Day\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"Instant Hole Kit\",\n",
        "                    \"quantity\": 3,\n",
        "                    \"unit_price\": 45.00,\n",
        "                    \"delivery_option\": \"Standard\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"TNT High Explosives x24\",\n",
        "                    \"quantity\": 1,\n",
        "                    \"unit_price\": 120.00,\n",
        "                    \"delivery_option\": \"Fast\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"Super Magnet (XL)\",\n",
        "                    \"quantity\": 1,\n",
        "                    \"unit_price\": 150.00,\n",
        "                    \"delivery_option\": \"Fast\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"Rocket-Powered Roller skates\",\n",
        "                    \"quantity\": 2,\n",
        "                    \"unit_price\": 79.99,\n",
        "                    \"delivery_option\": \"Next Day\",\n",
        "                },\n",
        "            ],\n",
        "        },\n",
        "    },\n",
        "    {\n",
        "        \"image_url\": \"https://storage.googleapis.com/github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/EF0004.jpg\",\n",
        "        \"image_uri\": \"gs://github-repo/generative-ai/gemini/evaluation/evaluate_gemini_structured_output/EF0004.jpg\",\n",
        "        \"image_type\": \"image/jpeg\",\n",
        "        \"image_name\": \"EF0004.jpg\",\n",
        "        \"reference\": {  # The Ground Truth\n",
        "            \"order_number\": \"EF0004\",\n",
        "            \"order_date\": \"2025-10-26\",\n",
        "            \"customer_name\": \"Elmer J. Fudd\",\n",
        "            \"customer_address\": \"Happy Hunter's Hollow, Looney Tune Forest, CA\",\n",
        "            \"line_items\": [\n",
        "                {\n",
        "                    \"item_description\": \"Silent Sneak Shoes\",\n",
        "                    \"quantity\": 1,\n",
        "                    \"unit_price\": 35.99,\n",
        "                    \"delivery_option\": \"Standard\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"Invisible Rabbit Trap\",\n",
        "                    \"quantity\": 2,\n",
        "                    \"unit_price\": 75.00,\n",
        "                    \"delivery_option\": \"Standard\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"Giant Butterfly Net\",\n",
        "                    \"quantity\": 1,\n",
        "                    \"unit_price\": 49.50,\n",
        "                    \"delivery_option\": \"Fast\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"Instant Camouflage Kit\",\n",
        "                    \"quantity\": 3,\n",
        "                    \"unit_price\": 65.00,\n",
        "                    \"delivery_option\": \"Next Day\",\n",
        "                },\n",
        "                {\n",
        "                    \"item_description\": \"Repellent Spray\",\n",
        "                    \"quantity\": 4,\n",
        "                    \"unit_price\": 29.99,\n",
        "                    \"delivery_option\": \"Next Day\",\n",
        "                },\n",
        "            ],\n",
        "        },\n",
        "    },\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mCjahTXbx9c2"
      },
      "outputs": [],
      "source": [
        "# Display the images using their public URLs\n",
        "\n",
        "for image in images:\n",
        "    print(image[\"image_name\"])\n",
        "    display(Image(url=image[\"image_url\"], height=800))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f8gOpub_1GYB"
      },
      "source": [
        "These are mock order forms for *Acme Corporation*, for customers to order various products, and select a delivery option for each; either \"Standard\", \"Fast\" or \"Next Day\".\n",
        "\n",
        "We will use this form to evaluate the performance of Gemini."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TWKVr7mmjxy0"
      },
      "source": [
        "## Extract the data using Gemini\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X0UJShmGMv3h"
      },
      "source": [
        "### Select the models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3FWWciVnjxy0"
      },
      "outputs": [],
      "source": [
        "# Define which models to compare\n",
        "\n",
        "models = [\n",
        "    # Gemini 2.0 family\n",
        "    \"gemini-2.0-flash\",\n",
        "    # Gemini 2.5 family\n",
        "    \"gemini-2.5-flash\",\n",
        "    \"gemini-2.5-pro\",\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qiQ4jCWNM3lj"
      },
      "source": [
        "### Define the prompt and schema"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cQScA9Wujxy0"
      },
      "outputs": [],
      "source": [
        "# Define the prompt and the structured output schema\n",
        "\n",
        "prompt = \"\"\"\n",
        "    Analyze the attached scanned form and extract the information in the table in accordance with the schema.\n",
        "\n",
        "    Provide the output in a clean JSON format.\n",
        "\n",
        "    If any date field is formatted ambiguously, assume the dates are in dd/mm/yyyy format.\n",
        "\n",
        "    If a field is blank, illegible, or cannot be found, return null for its value.\n",
        "\n",
        "    If there are blank rows, do not include them in the output.\n",
        "\n",
        "    If there is no image attached, return null for all fields.\n",
        "\n",
        "\"\"\"\n",
        "\n",
        "# Use structured output to ensure well formatted and consistent JSON output\n",
        "\n",
        "schema = {\n",
        "    \"type\": \"object\",\n",
        "    \"properties\": {\n",
        "        \"order_number\": {\"type\": \"string\"},\n",
        "        \"order_date\": {\n",
        "            \"type\": \"string\",\n",
        "            \"format\": \"date\",  # Note: Enforces a full date output in the RFC 3339 format (\"YYYY-MM-DD\")\n",
        "        },\n",
        "        \"customer_name\": {\"type\": \"string\"},\n",
        "        \"customer_address\": {\"type\": \"string\"},\n",
        "        \"line_items\": {\n",
        "            \"type\": \"array\",\n",
        "            \"items\": {\n",
        "                \"type\": \"object\",\n",
        "                \"properties\": {\n",
        "                    \"item_description\": {\"type\": \"string\"},\n",
        "                    \"quantity\": {\"type\": \"integer\"},\n",
        "                    \"unit_price\": {\"type\": \"number\"},\n",
        "                    \"delivery_option\": {  # Note: We do not tell Gemini how to interpret the checkboxes as \"Standard\", \"Fast\" or \"Next Day\"\n",
        "                        \"type\": \"string\"\n",
        "                    },\n",
        "                },\n",
        "            },\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "generate_content_config = GenerateContentConfig(\n",
        "    response_mime_type=\"application/json\",\n",
        "    response_schema=schema,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F57ILmc0M8ZE"
      },
      "source": [
        "### Run the prompt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cCqG3xE_jxy0"
      },
      "outputs": [],
      "source": [
        "# Run the prompt for each model in `models` and each image in `images`, storing the output in `gemini_response`\n",
        "\n",
        "gemini_response = {}\n",
        "run_id = notebook_utils.generate_uuid(8)\n",
        "\n",
        "\n",
        "for image_info in images:\n",
        "    image = Part.from_uri(\n",
        "        file_uri=image_info[\"image_uri\"], mime_type=image_info[\"image_type\"]\n",
        "    )\n",
        "    image_name = image_info[\"image_name\"]\n",
        "\n",
        "    gemini_response[image_name] = {}\n",
        "\n",
        "    for model in models:\n",
        "        run_name = f\"{run_id}-{model}-{image_name}\"\n",
        "\n",
        "        response = client.models.generate_content(\n",
        "            model=model, contents=[prompt, image], config=generate_content_config\n",
        "        )\n",
        "\n",
        "        response_json = json.dumps(response.parsed, indent=4)\n",
        "\n",
        "        print(\"----------------------------------\")\n",
        "        print(f\"{run_name}: \\n{response_json}\")\n",
        "        gemini_response[image_name][model] = response_json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oUNVUz6nNHQE"
      },
      "source": [
        "## Perform the Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdvJRUWRNGHE"
      },
      "source": [
        "### Prepare the evaluation dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rajaEV5i6auy"
      },
      "source": [
        "Now we have the outputs from the Gemini models we can run the evaulation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hUGFuYawEJYX"
      },
      "outputs": [],
      "source": [
        "# Create the Evaluation Dataset\n",
        "\n",
        "eval_dataset_rows = []\n",
        "for image_info in images:\n",
        "    image_name = image_info[\"image_name\"]\n",
        "    image_uri = image_info[\"image_uri\"]\n",
        "    image_type = image_info[\"image_type\"]\n",
        "    reference_str = json.dumps(\n",
        "        image_info[\"reference\"], indent=4\n",
        "    )  # Convert the reference (ground truth) to pretty-printed JSON\n",
        "\n",
        "    if image_name in gemini_response:\n",
        "        models_data = gemini_response[image_name]\n",
        "        for model_name, response_text in models_data.items():\n",
        "            eval_dataset_rows.append(\n",
        "                {\n",
        "                    \"model\": model_name,\n",
        "                    \"prompt\": prompt,  # The same prompt is used for all Gemini calls\n",
        "                    \"image\": image_name,\n",
        "                    \"reference\": reference_str,\n",
        "                    \"response\": response_text,\n",
        "                    \"differences\": DeepDiff(\n",
        "                        json.loads(reference_str), json.loads(response_text)\n",
        "                    ).pretty(),  # Uses the `deepdiff` library for identifying the differences between the response and the references\n",
        "                }\n",
        "            )\n",
        "\n",
        "eval_dataset = pd.DataFrame(eval_dataset_rows)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yrTqbS6NAKHk"
      },
      "source": [
        "This evaluation data set now contains the reference (ground truth) and response for each combination of model and image."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E6ByFZhujxy1"
      },
      "source": [
        "### Define custom metrics for JSON schema validation and accuracy"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W2v2WdYbjxy1"
      },
      "outputs": [],
      "source": [
        "# Define a custom evaluation metric to assess whether the response complies with the schema\n",
        "\n",
        "\n",
        "def is_valid_schema(instance: dict[str, str]) -> dict[str, bool]:\n",
        "    \"\"\"Return 1 if the response complies with the schema, 0 if not.\"\"\"\n",
        "    response = instance[\"response\"]\n",
        "\n",
        "    try:\n",
        "        validate(instance=json.loads(response), schema=schema)\n",
        "    except Exception:\n",
        "        return {\"valid_schema\": False}\n",
        "\n",
        "    return {\"valid_schema\": True}\n",
        "\n",
        "\n",
        "valid_schema = CustomMetric(name=\"valid_schema\", metric_function=is_valid_schema)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H76MFhlcy5lA"
      },
      "outputs": [],
      "source": [
        "# Define a `CustomMetric` for accuracy using the `deepdiff` library\n",
        "\n",
        "\n",
        "def calculate_accuracy(instance: dict[str, str]) -> dict[str, float]:\n",
        "    ref_json_string = instance[\"reference\"]\n",
        "    resp_json_string = instance[\"response\"]\n",
        "\n",
        "    try:\n",
        "        reference_data = json.loads(ref_json_string)\n",
        "        response_data = json.loads(resp_json_string)\n",
        "    except json.JSONDecodeError:\n",
        "        # If JSON is invalid or parsing fails, return 0 accuracy\n",
        "        return {\"accuracy\": 0.0}\n",
        "\n",
        "    # Use the deepdiff library to calculate the difference between the response and the reference (0 = exact match, 1 = )\n",
        "    deep_distance = DeepDiff(\n",
        "        reference_data, response_data, ignore_order=True, get_deep_distance=True\n",
        "    ).get(\"deep_distance\")\n",
        "\n",
        "    accuracy = 1.0 - deep_distance\n",
        "\n",
        "    return {\"accuracy\": accuracy}\n",
        "\n",
        "\n",
        "accuracy = CustomMetric(name=\"accuracy\", metric_function=calculate_accuracy)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fXdy-2UVjxy1"
      },
      "source": [
        "### Define `EvalTask` & Experiment"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gRoVmM2gjxy1"
      },
      "outputs": [],
      "source": [
        "# Define the evaluation task\n",
        "\n",
        "extraction_eval_task = EvalTask(\n",
        "    dataset=eval_dataset,\n",
        "    metrics=[\n",
        "        \"exact_match\",  # Exact match will only be 1 if the response is perfectly accurate, with no allowance for inconsistent JSON formatting. Hence, the custom `accuracy` metric is the better metric.\n",
        "        valid_schema,\n",
        "        accuracy,\n",
        "    ],\n",
        "    experiment=EXPERIMENT_NAME,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pj69SXdqNdLC"
      },
      "source": [
        "### Run the Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e6KhR-uGjxy1"
      },
      "outputs": [],
      "source": [
        "# Define the experiment & experiment run\n",
        "run_id = notebook_utils.generate_uuid(8)\n",
        "\n",
        "experiment_run_name = f\"eval-{run_id}\"\n",
        "\n",
        "eval_result = extraction_eval_task.evaluate(experiment_run_name=experiment_run_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LXmjQ4vkNikd"
      },
      "source": [
        "### Display the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dQvVqjfrjxy1"
      },
      "outputs": [],
      "source": [
        "notebook_utils.display_eval_result(eval_result=eval_result)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tQMjObguZA-I"
      },
      "source": [
        "## Analyse the results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8-Lj96MvOqW4"
      },
      "source": [
        "### Use Gemini to analyse the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oS2LVREz1zOF"
      },
      "outputs": [],
      "source": [
        "# Let's get Gemini to analyse the results.\n",
        "\n",
        "# Prepare the prompt for Gemini 2.5 Flash to summarize and analyze the results\n",
        "summary_prompt = \"\"\"\n",
        "Analyze the following experiment results comparing Gemini models for extracted data from a scanned form.\n",
        "The results include a summary table with overall metrics and row-based metrics, as well as the specific differences between the extracted data and the reference (ground truth).\n",
        "\n",
        "Summarize the performance of each model based on the metrics provided (valid_schema, accuracy) from the summary table.\n",
        "Analyze the detailed differences to understand the *types* of errors and mismatches occurring for each model.\n",
        "Identify which models performed best and worst for each metric and based on the detailed error analysis.\n",
        "Draw conclusions about the strengths and weaknesses of Gemini models for this specific tabular data extraction task, considering both the overall accuracy and the nature of the errors.\n",
        "Consider the different versions of Gemini and how their performance varies.\n",
        "Provide a clear and concise summary of the overall results, followed by key conclusions supported by observations from the detailed comparison.\n",
        "\n",
        "Experiment Results Summary Table:\n",
        "\"\"\"\n",
        "\n",
        "# Convert the evaluation results summary and row-based metrics to a string format\n",
        "# Assuming eval_result has a structure that can be converted to a readable string\n",
        "try:\n",
        "    # This will likely involve converting the DataFrames within eval_result to string\n",
        "    eval_result_string = str(eval_result)\n",
        "except Exception as e:\n",
        "    eval_result_string = f\"Could not convert evaluation results to string: {e}\"\n",
        "    print(eval_result_string)\n",
        "\n",
        "\n",
        "# Concatenate the prompt and the summary table results\n",
        "full_prompt = summary_prompt + eval_result_string\n",
        "\n",
        "# Use Gemini 2.5 Flash to analyze the results\n",
        "try:\n",
        "    # Generate the response\n",
        "    response = client.models.generate_content(\n",
        "        model=\"gemini-2.5-flash\", contents=full_prompt\n",
        "    )\n",
        "\n",
        "    # Display the summary and analysis from Gemini\n",
        "    display(Markdown(response.text))\n",
        "\n",
        "except Exception as e:\n",
        "    print(f\"An error occurred while calling Gemini: {e}\")\n",
        "    print(\n",
        "        \"Please ensure you have access to Gemini 2.5 Flash and your project/location settings are correct.\"\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5xRAqAOCpk3C"
      },
      "source": [
        "## Conclusions\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TXVk6VeKpolc"
      },
      "source": [
        "This notebook has shown how to use the Gen AI Evaluation Service to evaluate Gemini's Structured Output, for a document processing task.\n",
        "\n",
        "It uses a \"bring your own response\" approach and uses custom `valid_schema` and `accuracy` metrics as well as the `exact_match` metric.\n",
        "\n",
        "It also does a deep \"field-wise\" comparison of the responses to understand inaccuracies, and uses Gemini to summarise and analyse the results."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "evaluate_gemini_structured_output.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
