{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Using open autorater for running evaluations with Vertex AI Gen AI Evaluation\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fopen-models%2Fevaluation%2Fvertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/open-models/evaluation/vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84f0f73a0f76"
      },
      "source": [
        "| Author(s) |\n",
        "| --- |\n",
        "| [Ivan Nardini](https://github.com/inardini) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook demostrates how to evaluate the performance of a Large Language Model (LLM) using the Vertex AI Generative AI Evaluation service. Specifically, you will learn how to leverage an open judge model deployed on Vertex AI, such as `AtlaAI/Selene-1-Mini-Llama-3.1-8B`, to evaluate responses generated by another LLM against predefined criteria.\n",
        "\n",
        "By following this tutorial, you will perform the following key steps:\n",
        "\n",
        "- **Deploy Judge Model:** Upload and deploy the open-source Selene model to a Vertex AI Endpoint to serve as the autorater.\n",
        "- **Test Prediction:** Send a sample request to the deployed Selene model to ensure it's operational.\n",
        "- **Prepare Data:** Load or define an evaluation dataset containing prompts, reference answers, model responses, and corresponding human ratings.\n",
        "- **Define Custom Metric:** Create a custom evaluation metric (e.g., 'Completeness') with a detailed prompt template and scoring rubric for the autorater.\n",
        "- **Run Evaluation:** Execute the Vertex AI evaluation task, instructing the deployed Selene model to score the target LLM's responses based on the custom metric.\n",
        "- **Evaluate the Autorater:** Compare the autorater's scores against the human ratings provided in the dataset to assess the autorater's alignment with human judgment (meta-evaluation).\n",
        "- **Visualize Results:** Generate plots (distributions, confusion matrix, scatter plot) to visually analyze the agreement between the autorater and human ratings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --quiet google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "# import sys\n",
        "\n",
        "# if \"google.colab\" in sys.modules:\n",
        "#     from google.colab import auth\n",
        "\n",
        "#     auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f9af3e57f89a"
      },
      "source": [
        "### Authenticate your Hugging Face account\n",
        "\n",
        "Authenticate with Hugging Face Hub using the `interpreter_login` function from the `huggingface_hub` library, which will prompt you to enter a Hugging Face access token. This token authorizes the notebook to download the model artifacts and tokenizer.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8d836e0210fe"
      },
      "outputs": [],
      "source": [
        "from huggingface_hub import interpreter_login\n",
        "\n",
        "interpreter_login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c71a4314c250"
      },
      "source": [
        "Read more about [Hugging Face Security](https://huggingface.co/docs/hub/en/security), specifically about [Hugging Face User Access Tokens](https://huggingface.co/docs/hub/en/security-tokens)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "# Use the environment variable if the user doesn't provide Project ID.\n",
        "import os\n",
        "\n",
        "import vertexai\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
        "\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "LOCATION = os.environ.get(\"GOOGLE_CLOUD_REGION\", \"us-central1\")\n",
        "\n",
        "BUCKET_NAME = \"[your-bucket-name]\"  # @param {type: \"string\", placeholder: \"[your-bucket-name]\", isTemplate: true}\n",
        "\n",
        "BUCKET_URI = f\"gs://{BUCKET_NAME}\"\n",
        "\n",
        "!gsutil mb -l {LOCATION} {BUCKET_URI}\n",
        "\n",
        "EXPERIMENT_NAME = \"eval-open-judge\"  # @param {type:\"string\"}\n",
        "\n",
        "vertexai.init(\n",
        "    project=PROJECT_ID,\n",
        "    location=LOCATION,\n",
        "    staging_bucket=BUCKET_URI,\n",
        "    experiment=EXPERIMENT_NAME,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "haaUJ5VaWajl"
      },
      "source": [
        "## Import libraries\n",
        "\n",
        "Imports the necessary Python libraries used throughout the notebook.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kCp1oCOyWfZe"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from typing import Any\n",
        "\n",
        "from IPython.display import Markdown, display\n",
        "from google.cloud import aiplatform\n",
        "from huggingface_hub import get_token\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import plotly.graph_objects as go\n",
        "from plotly.subplots import make_subplots\n",
        "from transformers import AutoTokenizer\n",
        "from vertexai.preview.evaluation import AutoraterConfig, EvalTask, PointwiseMetric\n",
        "from vertexai.preview.evaluation.autorater_utils import evaluate_autorater"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JySe_tubAp9O"
      },
      "source": [
        "## Define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nL5kyMVHAsf2"
      },
      "outputs": [],
      "source": [
        "JITTER_AMOUNT = 0.1\n",
        "PLOTLY_RENDERER = \"colab\"\n",
        "DEFAULT_HUMAN_RATING_COL = \"human_rating\"\n",
        "DEFAULT_SCORE_COL = \"score\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvVKEpROoXW5"
      },
      "source": [
        "## Define helpers\n",
        "\n",
        "Defines some helpers function to format user requests and plot an evaluation report.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kR0Fh3OMEEqh"
      },
      "outputs": [],
      "source": [
        "def format_user_content(user_content, tokenizer, **kwargs):\n",
        "    \"\"\"\n",
        "    Applies tokenizer.apply_chat_template to user content string.\n",
        "    Assumes user_content is the text for the 'user' role.\n",
        "    \"\"\"\n",
        "\n",
        "    message = [\n",
        "        {\"role\": \"user\", \"content\": user_content},\n",
        "    ]\n",
        "    kwargs.setdefault(\"tokenize\", False)\n",
        "    kwargs.setdefault(\"add_generation_prompt\", True)\n",
        "\n",
        "    try:\n",
        "        formatted_input = tokenizer.apply_chat_template(message, **kwargs)\n",
        "        return formatted_input\n",
        "    except Exception as e:\n",
        "        print(\n",
        "            f\"Error applying chat template to content: '{user_content[:50]}...'. Error: {e}\"\n",
        "        )\n",
        "        return None\n",
        "\n",
        "\n",
        "def prepare_dataframe(\n",
        "    raw_data: Any,\n",
        "    human_col_name: str = DEFAULT_HUMAN_RATING_COL,\n",
        "    score_col_name: str = DEFAULT_SCORE_COL,\n",
        ") -> pd.DataFrame:\n",
        "    \"\"\"\n",
        "    Prepares the DataFrame from raw data, renames columns, and converts types.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        if isinstance(raw_data, pd.DataFrame):\n",
        "            df = raw_data.copy()\n",
        "            if len(df.columns) >= 2:\n",
        "                # If it's a DataFrame, check if desired columns exist, else use first two\n",
        "                if human_col_name not in df.columns or score_col_name not in df.columns:\n",
        "                    original_cols = df.columns\n",
        "                    df = df[\n",
        "                        [original_cols[0], original_cols[1]]\n",
        "                    ].copy()  # Use first two\n",
        "                    df.columns = [human_col_name, score_col_name]\n",
        "                else:\n",
        "                    # Ensure we only keep the needed columns if more exist\n",
        "                    df = df[[human_col_name, score_col_name]].copy()\n",
        "            else:\n",
        "                print(\n",
        "                    f\"Warning: Input DataFrame has < 2 columns. Expected at least '{human_col_name}' and '{score_col_name}'.\"\n",
        "                )\n",
        "                return pd.DataFrame(columns=[human_col_name, score_col_name])\n",
        "        else:\n",
        "            df = pd.DataFrame(raw_data)\n",
        "            if df.empty:\n",
        "                print(\"Warning: Created empty DataFrame from raw_data.\")\n",
        "                return pd.DataFrame(columns=[human_col_name, score_col_name])\n",
        "\n",
        "            if len(df.columns) >= 2:\n",
        "                df = df.iloc[\n",
        "                    :, :2\n",
        "                ]  # Assume first two columns are human rating and score\n",
        "                df.columns = [human_col_name, score_col_name]  # Rename for simplicity\n",
        "            else:\n",
        "                print(\n",
        "                    f\"Warning: DataFrame from raw_data has < 2 columns. Cannot set '{human_col_name}' and '{score_col_name}'.\"\n",
        "                )\n",
        "                return pd.DataFrame(columns=[human_col_name, score_col_name])\n",
        "\n",
        "        df[human_col_name] = pd.to_numeric(df[human_col_name], errors=\"coerce\")\n",
        "        df[score_col_name] = pd.to_numeric(df[score_col_name], errors=\"coerce\")\n",
        "        df.dropna(subset=[human_col_name, score_col_name], inplace=True)\n",
        "        df[human_col_name] = df[human_col_name].astype(float)\n",
        "        df[score_col_name] = df[score_col_name].astype(float)\n",
        "        return df\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error preparing DataFrame: {e}\")\n",
        "        return pd.DataFrame(columns=[human_col_name, score_col_name])  # Return empty df\n",
        "\n",
        "\n",
        "def extract_completeness_metrics(\n",
        "    metrics_data: list[dict[str, Any]] | None,\n",
        ") -> tuple[list[list[Any]] | None, list[str] | None, list[float] | None]:\n",
        "    \"\"\"\n",
        "    Extracts confusion matrix info from metrics data (expected at index 0).\n",
        "    \"\"\"\n",
        "    if not metrics_data or not isinstance(metrics_data, list) or len(metrics_data) == 0:\n",
        "        print(\"Warning: Metrics data is empty or not a list.\")\n",
        "        return None, None, None\n",
        "\n",
        "    try:\n",
        "        completeness_metrics = metrics_data[0]\n",
        "        if (\n",
        "            \"confusion_matrix\" not in completeness_metrics\n",
        "            or \"confusion_matrix_labels\" not in completeness_metrics\n",
        "        ):\n",
        "            print(\n",
        "                \"Warning: 'confusion_matrix' or 'confusion_matrix_labels' not in first metrics item.\"\n",
        "            )\n",
        "            return None, None, None\n",
        "\n",
        "        cm = completeness_metrics[\"confusion_matrix\"]\n",
        "        cm_labels = completeness_metrics[\"confusion_matrix_labels\"]\n",
        "        cm_labels_numeric = [float(cl) for cl in cm_labels]\n",
        "        return cm, cm_labels, cm_labels_numeric\n",
        "    except (KeyError, IndexError, ValueError, TypeError) as e:\n",
        "        print(f\"Warning: Could not extract confusion matrix info from metrics: {e}\")\n",
        "        return None, None, None\n",
        "\n",
        "\n",
        "def plot_distribution_comparison(\n",
        "    df: pd.DataFrame,\n",
        "    human_col: str = DEFAULT_HUMAN_RATING_COL,\n",
        "    score_col: str = DEFAULT_SCORE_COL,\n",
        ") -> go.Figure:\n",
        "    \"\"\"Generates bar charts comparing distributions of human ratings and model scores.\"\"\"\n",
        "    human_counts = df[human_col].value_counts().sort_index()\n",
        "    score_counts = df[score_col].value_counts().sort_index()\n",
        "\n",
        "    fig = make_subplots(\n",
        "        rows=1,\n",
        "        cols=2,\n",
        "        subplot_titles=(\"Human Rating Distribution\", \"Model Score Distribution\"),\n",
        "    )\n",
        "\n",
        "    fig.add_trace(\n",
        "        go.Bar(\n",
        "            x=human_counts.index,\n",
        "            y=human_counts.values,\n",
        "            name=\"Human Rating\",\n",
        "            marker_color=\"indianred\",\n",
        "        ),\n",
        "        row=1,\n",
        "        col=1,\n",
        "    )\n",
        "\n",
        "    fig.add_trace(\n",
        "        go.Bar(\n",
        "            x=score_counts.index,\n",
        "            y=score_counts.values,\n",
        "            name=\"Model Score\",\n",
        "            marker_color=\"lightsalmon\",\n",
        "        ),\n",
        "        row=1,\n",
        "        col=2,\n",
        "    )\n",
        "\n",
        "    fig.update_layout(\n",
        "        title_text=\"Distribution of Human Ratings vs. Model Scores\",\n",
        "        bargap=0.2,\n",
        "        xaxis1_title=\"Rating Value\",\n",
        "        yaxis1_title=\"Count\",\n",
        "        xaxis2_title=\"Score Value\",\n",
        "        yaxis2_title=\"Count\",\n",
        "        xaxis1_type=\"category\",\n",
        "        xaxis2_type=\"category\",\n",
        "        xaxis1=dict(\n",
        "            categoryorder=\"array\", categoryarray=sorted(human_counts.index.unique())\n",
        "        ),\n",
        "        xaxis2=dict(\n",
        "            categoryorder=\"array\", categoryarray=sorted(score_counts.index.unique())\n",
        "        ),\n",
        "        height=400,\n",
        "    )\n",
        "    return fig\n",
        "\n",
        "\n",
        "def plot_confusion_matrix(\n",
        "    cm: list[list[Any]] | None, cm_labels: list[str] | None\n",
        ") -> go.Figure | None:\n",
        "    \"\"\"Generates a heatmap for the confusion matrix.\"\"\"\n",
        "    if cm is None or cm_labels is None:\n",
        "        print(\"Skipping confusion matrix plot: missing data.\")\n",
        "        return None\n",
        "\n",
        "    fig = go.Figure(\n",
        "        data=go.Heatmap(\n",
        "            z=cm,\n",
        "            x=cm_labels,\n",
        "            y=cm_labels,\n",
        "            hoverongaps=False,\n",
        "            colorscale=\"Blues\",\n",
        "            text=cm,\n",
        "            texttemplate=\"%{text}\",\n",
        "            zmin=0,\n",
        "        )\n",
        "    )\n",
        "\n",
        "    fig.update_layout(\n",
        "        title=\"Confusion Matrix: Human Rating vs. Model Score (Completeness)\",\n",
        "        xaxis_title=\"Predicted (Model Score)\",\n",
        "        yaxis_title=\"True (Human Rating)\",\n",
        "        yaxis=dict(type=\"category\", categoryorder=\"array\", categoryarray=cm_labels),\n",
        "        xaxis=dict(type=\"category\", categoryorder=\"array\", categoryarray=cm_labels),\n",
        "        height=600,\n",
        "        width=600,\n",
        "    )\n",
        "    return fig\n",
        "\n",
        "\n",
        "def plot_jitter_scatter(\n",
        "    df: pd.DataFrame,\n",
        "    cm_labels: list[str] | None,\n",
        "    cm_labels_numeric: list[float] | None,\n",
        "    human_col: str = DEFAULT_HUMAN_RATING_COL,\n",
        "    score_col: str = DEFAULT_SCORE_COL,\n",
        ") -> go.Figure:\n",
        "    \"\"\"Generates a jitter scatter plot comparing individual scores and ratings.\"\"\"\n",
        "    df_jitter = df[[human_col, score_col]].copy()\n",
        "\n",
        "    if cm_labels is None or cm_labels_numeric is None:\n",
        "        print(\n",
        "            \"Using data range for jitter plot axes due to missing confusion matrix labels.\"\n",
        "        )\n",
        "        min_val: float = (\n",
        "            min(df_jitter[human_col].min(), df_jitter[score_col].min()) - 0.5\n",
        "        )\n",
        "        max_val: float = (\n",
        "            max(df_jitter[human_col].max(), df_jitter[score_col].max()) + 0.5\n",
        "        )\n",
        "        plot_range = [min_val, max_val]\n",
        "        tick_vals = sorted(\n",
        "            df_jitter[human_col].unique()\n",
        "        )  # Use unique human ratings for ticks if available\n",
        "        tick_text = [str(int(v)) if v == int(v) else str(v) for v in tick_vals]\n",
        "    else:\n",
        "        plot_range = [min(cm_labels_numeric) - 0.5, max(cm_labels_numeric) + 0.5]\n",
        "        tick_vals = cm_labels_numeric\n",
        "        tick_text = cm_labels\n",
        "\n",
        "    # Add jitter\n",
        "    df_jitter[f\"{human_col}_jitter\"] = df_jitter[human_col] + np.random.uniform(\n",
        "        -JITTER_AMOUNT, JITTER_AMOUNT, size=len(df_jitter)\n",
        "    )\n",
        "    df_jitter[f\"{score_col}_jitter\"] = df_jitter[score_col] + np.random.uniform(\n",
        "        -JITTER_AMOUNT, JITTER_AMOUNT, size=len(df_jitter)\n",
        "    )\n",
        "\n",
        "    fig = go.Figure()\n",
        "\n",
        "    fig.add_trace(\n",
        "        go.Scatter(\n",
        "            x=df_jitter[f\"{score_col}_jitter\"],\n",
        "            y=df_jitter[f\"{human_col}_jitter\"],\n",
        "            mode=\"markers\",\n",
        "            marker=dict(\n",
        "                color=\"rgba(0, 100, 200, 0.7)\",\n",
        "                size=10,\n",
        "                line=dict(width=1, color=\"DarkSlateGrey\"),\n",
        "            ),\n",
        "            text=[\n",
        "                f\"HR: {hr:.1f}, Score: {s:.1f}\"\n",
        "                for hr, s in zip(df_jitter[human_col], df_jitter[score_col])\n",
        "            ],  # Format hover text\n",
        "            hoverinfo=\"text\",\n",
        "            name=\"Ratings\",\n",
        "        )\n",
        "    )\n",
        "\n",
        "    # Add ideal alignment line (y=x)\n",
        "    fig.add_trace(\n",
        "        go.Scatter(\n",
        "            x=plot_range,\n",
        "            y=plot_range,\n",
        "            mode=\"lines\",\n",
        "            name=\"Ideal Alignment (Score = Human Rating)\",\n",
        "            line=dict(color=\"red\", dash=\"dash\"),\n",
        "        )\n",
        "    )\n",
        "\n",
        "    fig.update_layout(\n",
        "        title=\"Model Score vs. Human Rating (with Jitter)\",\n",
        "        xaxis_title=\"Model Score (Jittered)\",\n",
        "        yaxis_title=\"Human Rating (Jittered)\",\n",
        "        xaxis=dict(\n",
        "            range=plot_range, tickvals=list(tick_vals), ticktext=tick_text\n",
        "        ),  # tickvals expects list\n",
        "        yaxis=dict(range=plot_range, tickvals=list(tick_vals), ticktext=tick_text),\n",
        "        width=600,\n",
        "        height=600,\n",
        "        showlegend=True,\n",
        "        hovermode=\"closest\",\n",
        "    )\n",
        "    return fig\n",
        "\n",
        "\n",
        "def run_visual_analysis(\n",
        "    df_data: Any,\n",
        "    metrics: list[dict[str, Any]] | None,\n",
        "    human_col_name: str = DEFAULT_HUMAN_RATING_COL,\n",
        "    score_col_name: str = DEFAULT_SCORE_COL,\n",
        ") -> None:\n",
        "    \"\"\"\n",
        "    Runs the visual analysis comparing model scores and human ratings.\n",
        "    \"\"\"\n",
        "    display(Markdown(\"# Visual Analysis: Model Score vs. Human Rating Alignment \"))\n",
        "\n",
        "    # 1. Prepare Data\n",
        "    df = prepare_dataframe(\n",
        "        df_data, human_col_name=human_col_name, score_col_name=score_col_name\n",
        "    )\n",
        "    cm, cm_labels, cm_labels_numeric = extract_completeness_metrics(metrics)\n",
        "\n",
        "    # Check if DataFrame creation failed\n",
        "    if df.empty:\n",
        "        display(\n",
        "            Markdown(\n",
        "                f\"Could not create valid DataFrame with columns '{human_col_name}' and '{score_col_name}' from input data. Stopping analysis.\"\n",
        "            )\n",
        "        )\n",
        "        return\n",
        "\n",
        "    if human_col_name not in df.columns or score_col_name not in df.columns:\n",
        "        display(\n",
        "            Markdown(\n",
        "                f\"Expected columns '{human_col_name}' and '{score_col_name}' not found in DataFrame. Stopping analysis.\"\n",
        "            )\n",
        "        )\n",
        "        return\n",
        "\n",
        "    # 2. Visualization 1: Distribution Comparison\n",
        "    display(Markdown(\"## 1. Distributions Comparison\"))\n",
        "    fig_dist: go.Figure | None = plot_distribution_comparison(\n",
        "        df, human_col=human_col_name, score_col=score_col_name\n",
        "    )\n",
        "    if fig_dist:\n",
        "        fig_dist.show(renderer=PLOTLY_RENDERER)\n",
        "        display(\n",
        "            Markdown(\"*Shows if the model score distribution mirrors human ratings.*\")\n",
        "        )\n",
        "    else:\n",
        "        display(Markdown(\"*Could not generate distribution comparison plot.*\"))\n",
        "    print(\"\\n\")\n",
        "\n",
        "    # 3. Visualization 2: Confusion Matrix Heatmap\n",
        "    display(Markdown(\"## 2. Confusion Matrix (Human vs. Model)\"))\n",
        "    fig_cm: go.Figure | None = plot_confusion_matrix(cm, cm_labels)\n",
        "    if fig_cm:\n",
        "        fig_cm.show(renderer=PLOTLY_RENDERER)\n",
        "        display(\n",
        "            Markdown(\n",
        "                \"*Visualizes agreement and disagreement between discrete human ratings and model scores. Ideal alignment is along the diagonal.*\"\n",
        "            )\n",
        "        )\n",
        "    else:\n",
        "        display(\n",
        "            Markdown(\n",
        "                \"*Confusion Matrix could not be generated (check metrics data or ensure it's at index 0).**\"\n",
        "            )\n",
        "        )\n",
        "    print(\"\\n\")\n",
        "\n",
        "    # 4. Visualization 3: Jitter Scatter Plot\n",
        "    display(Markdown(\"## 3. Score vs. Rating Alignment (Jitter Plot)\"))\n",
        "    fig_scatter: go.Figure | None = plot_jitter_scatter(\n",
        "        df,\n",
        "        cm_labels,\n",
        "        cm_labels_numeric,\n",
        "        human_col=human_col_name,\n",
        "        score_col=score_col_name,\n",
        "    )\n",
        "    if fig_scatter:\n",
        "        fig_scatter.show(renderer=PLOTLY_RENDERER)\n",
        "        display(\n",
        "            Markdown(\n",
        "                \"*Shows individual item alignment. Points close to the red dashed line indicate good agreement.*\"\n",
        "            )\n",
        "        )\n",
        "    else:\n",
        "        display(Markdown(\"*Could not generate jitter scatter plot.*\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3BOPydegEkP2"
      },
      "source": [
        "## Deploy your open judge model\n",
        "\n",
        "Deploying an open model means making it available as a callable API endpoint within Vertex AI.\n",
        "\n",
        "First, the `aiplatform.Model.upload(...)` function performs the registration of the model with Vertex AI, specifying its Hugging Face ID (`AtlaAI/Selene-1-Mini-Llama-3.1-8B`), the serving container image (a pre-built Text Generation Inference container), and necessary environment variables like your Hugging Face token."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "j0ekN7AP2IrR"
      },
      "outputs": [],
      "source": [
        "judge_model = aiplatform.Model.upload(\n",
        "    display_name=\"google--selene-1-mini-llama-3.1-8b\",\n",
        "    serving_container_image_uri=\"us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu124.2-4.ubuntu2204.py311\",\n",
        "    serving_container_environment_variables={\n",
        "        \"MODEL_ID\": \"AtlaAI/Selene-1-Mini-Llama-3.1-8B\",\n",
        "        \"NUM_SHARD\": \"1\",\n",
        "        \"HUGGING_FACE_HUB_TOKEN\": get_token(),\n",
        "    },\n",
        "    serving_container_ports=[8080],\n",
        ")\n",
        "judge_model.wait()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T4nhndYaCMVu"
      },
      "source": [
        "After the model is uploaded, `judge_model.deploy(...)` deploys it to a newly created Vertex AI Endpoint, specifying the required hardware (machine type, GPU type and count). This makes the Selene model ready to receive prediction requests, in this case, requests to evaluate LLM responses."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2qxLtEO22sgL"
      },
      "outputs": [],
      "source": [
        "deployed_judge_model = judge_model.deploy(\n",
        "    endpoint=aiplatform.Endpoint.create(\n",
        "        display_name=\"google--selene-1-mini-llama-3.1-8b-endpoint\"\n",
        "    ),\n",
        "    machine_type=\"g2-standard-4\",\n",
        "    accelerator_type=\"NVIDIA_L4\",\n",
        "    accelerator_count=1,\n",
        "    min_replica_count=1,\n",
        "    max_replica_count=1,\n",
        "    enable_access_logging=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tmuwCPShxso1"
      },
      "source": [
        "## Generate predictions\n",
        "\n",
        "You can verify that the deployed Selene model endpoint is active and responding correctly by loading the model's specific tokenizer, formatting a simple test message using the chat template, and then sending this formatted input to the endpoint via the `deployed_model.predict(...)` method.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J1YEakBU9Xgr"
      },
      "outputs": [],
      "source": [
        "tokenizer = AutoTokenizer.from_pretrained(\n",
        "    \"AtlaAI/Selene-1-Mini-Llama-3.1-8B\", token=get_token()\n",
        ")\n",
        "\n",
        "messages = [\n",
        "    {\"role\": \"user\", \"content\": \"I heard you can evaluate my responses?\"},\n",
        "]\n",
        "\n",
        "inputs = tokenizer.apply_chat_template(\n",
        "    messages,\n",
        "    tokenize=False,\n",
        "    add_generation_prompt=True,\n",
        ")\n",
        "\n",
        "prediction = deployed_judge_model.predict(\n",
        "    instances=[\n",
        "        {\n",
        "            \"inputs\": inputs,\n",
        "            \"parameters\": {\n",
        "                \"temperature\": 0,\n",
        "                \"max_new_tokens\": 512,\n",
        "            },\n",
        "        },\n",
        "    ]\n",
        ")\n",
        "print(prediction.predictions[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BnzsikoX6j8R"
      },
      "source": [
        "## Evaluate your LLM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cxXdWMu6fWb1"
      },
      "source": [
        "### Prepare the evaluation data\n",
        "\n",
        "To evaluate your LLM, you start by preparing your evaluation data as Pandas DataFrame. The dataframe contains lists for `user_input` (prompts), `ground_truth` (ideal answers), `assistant_response` (the actual responses from the LLM being evaluated), and `completeness/human_rating` (scores given by humans for one specific metric).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SaQ3xqNxhss7"
      },
      "outputs": [],
      "source": [
        "human_rated_dict = {\n",
        "    \"user_input\": [\n",
        "        \"Researchers at the Institute for Advanced Studies have developed a new type of solar panel that boasts a 5% increase in efficiency compared to current market leaders. The innovation lies in a novel perovskite crystal structure that is both more stable and better at capturing a wider spectrum of light. Commercial production is expected within three years.\",\n",
        "        \"Introducing the 'SilentStep' treadmill. Engineered with advanced noise-reduction technology, it allows for near-silent operation, perfect for apartment living or early morning workouts. It features 12 pre-set programs, a heart rate monitor, and folds easily for storage. Maximum user weight is 250 lbs.\",\n",
        "        \"This study investigated the effects of intermittent fasting (IF) versus daily caloric restriction (DCR) on metabolic markers in overweight adults over 12 weeks. Both groups achieved similar weight loss. However, the IF group showed significantly better improvements in insulin sensitivity and reduction in visceral fat compared to the DCR group, suggesting potential unique metabolic benefits beyond weight loss alone.\",\n",
        "        \"The old lighthouse stood sentinel on the cliff, its beam cutting through the thick fog rolling in from the sea. For generations, its light had guided ships safely to the harbor below. Elias, the keeper, felt the weight of that tradition as he climbed the winding stairs for his nightly duty, the rhythmic groan of the turning lens a familiar comfort.\",\n",
        "        \"The project planning meeting concluded with action items assigned. Marketing (Jane) to finalize competitor analysis by Friday. Engineering (Tom) to provide a prototype schematic by next Wednesday. Budget approval pending confirmation from Finance (Mr. Davies). Next sync meeting scheduled for Thursday, 10 AM.\",\n",
        "        \"To prepare the marinade, combine 1/4 cup soy sauce, 2 tablespoons honey, 1 tablespoon sesame oil, 2 minced garlic cloves, and 1 teaspoon grated ginger in a bowl. Whisk well. Add your protein (chicken, beef, or tofu) and ensure it's fully coated. Marinate for at least 30 minutes, or preferably 2 hours in the refrigerator.\",\n",
        "        \"The Library of Alexandria, in Egypt, was one of the largest and most significant libraries of the ancient world. Flourishing under the Ptolemaic dynasty, it was dedicated to the Muses, the nine goddesses of the arts. It functioned more as a research institution, attracting scholars from across the Hellenistic world, but its eventual destruction remains a subject of debate among historians.\",\n",
        "        \"A blockchain is a distributed, immutable ledger. Transactions are grouped into blocks, each cryptographically linked to the previous one using a hash. This chain structure, combined with decentralization across many computers, makes it extremely difficult to tamper with recorded data.\",\n",
        "        \"Deforestation in the Amazon rainforest continues to be a major environmental concern, primarily driven by cattle ranching and agriculture. This loss of forest cover contributes significantly to global carbon emissions and biodiversity loss. Recent satellite data indicates a slight decrease in the rate of deforestation compared to the previous year, but levels remain alarmingly high.\",\n",
        "        \"While the novel's premise was intriguing - a world where memories can be traded - the execution felt uneven. Character development was shallow, particularly for the protagonist, and the pacing dragged significantly in the middle third. However, the world-building details were imaginative and offered glimpses of a truly fascinating concept.\",\n",
        "    ],\n",
        "    \"ground_truth\": [\n",
        "        \"A new solar panel developed by institute researchers shows a 5% efficiency gain over current leaders due to a novel, stable perovskite structure capturing more light. Commercialization is expected in three years.\",\n",
        "        \"The 'SilentStep' treadmill offers near-silent operation suitable for shared spaces. It includes 12 programs, a heart rate monitor, easy folding for storage, and supports up to 250 lbs.\",\n",
        "        \"A 12-week study comparing intermittent fasting (IF) and daily caloric restriction (DCR) in overweight adults found similar weight loss, but IF led to significantly better insulin sensitivity and visceral fat reduction, indicating unique metabolic advantages.\",\n",
        "        \"An old lighthouse keeper, Elias, feels the weight of tradition as he tends the light that has guided ships through fog for generations, finding comfort in the familiar sounds of the lighthouse.\",\n",
        "        \"Meeting takeaways: Jane (Marketing) to complete competitor analysis by Friday; Tom (Engineering) to deliver prototype schematic by next Wednesday. Budget approval awaits Finance confirmation. Next meeting: Thursday, 10 AM.\",\n",
        "        \"Whisk together soy sauce, honey, sesame oil, minced garlic, and grated ginger for the marinade. Coat protein (chicken, beef, tofu) and marinate for at least 30 minutes (2 hours refrigerated recommended).\",\n",
        "        \"The ancient Library of Alexandria in Egypt, significant under the Ptolemaic dynasty and dedicated to the Muses, served as a major research hub attracting scholars. Its destruction is historically debated.\",\n",
        "        \"A blockchain is a distributed, unchangeable ledger where transactions are grouped into blocks cryptographically linked in a chain, making tampering difficult due to decentralization.\",\n",
        "        \"Amazon deforestation, mainly from cattle ranching and agriculture, remains a critical environmental issue contributing to carbon emissions and biodiversity loss. While recent data shows a slight decrease, the rate is still alarmingly high.\",\n",
        "        \"The novel's intriguing memory-trading premise suffered from uneven execution, with shallow character development and slow pacing, though its imaginative world-building was a highlight.\",\n",
        "    ],\n",
        "    \"assistant_response\": [\n",
        "        \"Researchers developed a new solar panel. It is 5% more efficient. It uses a new perovskite crystal structure. This structure is stable and captures more light. It should be commercially produced in three years.\",\n",
        "        \"This is the 'SilentStep' treadmill. It is quiet because of noise-reduction tech. Good for apartments. Has 12 programs. Has heart monitor. Folds. Max weight 250 lbs.\",\n",
        "        \"Overweight adults were studied for 12 weeks comparing intermittent fasting (IF) and daily caloric restriction (DCR). Weight loss was similar. IF group had better insulin sensitivity and less visceral fat than DCR group. IF might have unique benefits.\",\n",
        "        \"A lighthouse on a cliff guides ships in fog. Elias is the keeper. He climbs the stairs for his duty. He feels the tradition. The lens groans.\",\n",
        "        \"The meeting ended. Jane will do competitor analysis by Friday. Tom will provide a schematic next Wednesday. Finance needs to approve the budget. The next meeting is Thursday at 10 AM.\",\n",
        "        \"Make the marinade: mix 1/4c soy sauce, 2tbsp honey, 1tbsp sesame oil, 2 cloves garlic (minced), 1tsp ginger (grated). Whisk it. Put protein in. Cover it. Marinate 30+ minutes, better for 2 hours in fridge.\",\n",
        "        \"The Library of Alexandria was a big ancient library in Egypt. It was important during the Ptolemaic rule and dedicated to the Muses. Scholars came there to research. How it was destroyed is debated by historians.\",\n",
        "        \"Blockchain is like a shared digital book that cannot be changed easily. Information (transactions) goes into blocks. Blocks are linked using crypto hashes. Because it's spread out on many computers, changing data is very hard.\",\n",
        "        \"Deforestation in the Amazon is a big worry. Cattle and farming are main causes. It increases carbon emissions and hurts biodiversity. Satellites show the rate decreased slightly last year, but it's still very high.\",\n",
        "        \"The book had a cool idea about trading memories. But it wasn't done perfectly. Characters weren't deep, especially the main one. The middle part was slow. The world details were creative and showed a good concept.\",\n",
        "    ],\n",
        "    \"completeness/human_rating\": [\"4\", \"5\", \"5\", \"4\", \"5\", \"2\", \"4\", \"2\", \"5\", \"4\"],\n",
        "}\n",
        "\n",
        "human_rated_dict[\"user_input\"] = [\n",
        "    f\"Summarize the following text:\\n{p}\" for p in human_rated_dict[\"user_input\"]\n",
        "]\n",
        "human_rated_dataset = pd.DataFrame(human_rated_dict)\n",
        "human_rated_dataset.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v1w_Z96I-NGf"
      },
      "source": [
        "### Define custom metrics\n",
        "\n",
        "You define the criteria the autorater (Selene) will use to judge the LLM responses. This template instructs the autorater on its task, provides the scoring rubric (defining scores 1-5 for 'Completeness'), specifies the required output format (Reasoning and Result), and includes placeholders (`{user_input}`, `{assistant_response}`, etc.) that will be filled with data for each evaluation instance.\n",
        "\n",
        "This template is then converted into a `PointwiseMetric` from the Vertex AI SDK, giving the metric a name (`completeness`) and linking it to the prompt template."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5Nf5PeKg4POj"
      },
      "outputs": [],
      "source": [
        "completeness_prompt_template = \"\"\"You are tasked with evaluating a response based on a given instruction (which may contain an Input) and a scoring rubric and reference answer that serve as the evaluation standard. Provide a comprehensive feedback on the response quality strictly adhering to the scoring rubric, without any general evaluation. Follow this with a score between 1 and 5, referring to the scoring rubric. Avoid generating any additional opening, closing, or explanations.\n",
        "\n",
        "  Here are some rules of the evaluation:\n",
        "  (1) You should prioritize evaluating whether the response satisfies the provided rubric. The basis of your score should depend exactly on the rubric. However, the response does not need to explicitly address points raised in the rubric. Rather, evaluate the response based on the criteria outlined in the rubric.\n",
        "\n",
        "  Your reply should strictly follow this format:\n",
        "  **Reasoning:** <Your feedback>\n",
        "\n",
        "  **Result:** <an integer between 1 and 5>\n",
        "\n",
        "  Here is the data:\n",
        "\n",
        "  Instruction:\n",
        "  '''\n",
        "  {user_input}\n",
        "  '''\n",
        "\n",
        "  Response:\n",
        "  '''\n",
        "  {assistant_response}\n",
        "  '''\n",
        "\n",
        "  Score Rubrics:\n",
        "  Does the response provide a sufficient explanation? Comprehensiveness and thoroughness of the response should be considered, which depends on the breadth of\n",
        "  topics covered and the level of detail provided within each topic.\n",
        "  Score 1: The response doesn't include any specifics or examples to support the statements\n",
        "  made.\n",
        "  Score 2: The response does not provide sufficient details or supportive examples, requiring\n",
        "  a major effort to make the response more complete.\n",
        "  Score 3: It is a decent response, but the breadth and depth of the response are rather limited.\n",
        "  The details and examples used to substantiate the response may be insufficient.\n",
        "  Score 4: The response provides detailed explanations, but there is room for enhancement.\n",
        "  The response could be further improved by including more details and supportive examples.\n",
        "  Score 5: The response fully provides comprehensive explanations. It delves deep into the\n",
        "  topic, providing as much detail as possible, and it offers several examples to back up its\n",
        "  points.\n",
        "\n",
        "  Reference answer:\n",
        "  {ground_truth}\"\"\"\n",
        "\n",
        "completeness_messages = [\n",
        "    {\"role\": \"user\", \"content\": completeness_prompt_template},\n",
        "]\n",
        "\n",
        "completeness_prompt_template_fmt = tokenizer.apply_chat_template(\n",
        "    completeness_messages,\n",
        "    tokenize=False,\n",
        "    add_generation_prompt=True,\n",
        ")\n",
        "\n",
        "completeness = PointwiseMetric(\n",
        "    metric=\"completeness\",\n",
        "    metric_prompt_template=completeness_prompt_template_fmt,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RQ80NfSnAwPX"
      },
      "source": [
        "### Run evaluation\n",
        "\n",
        "To run an evaluation using Gen AI Evaluation service, you define an `EvalTask`. The `EvalTask` is configured with the evaluation `dataset`, the list of `metrics` to compute (our custom 'completeness' metric), the `experiment` name for tracking, and the `autorater_config`.\n",
        "\n",
        "The autorater configuration crucially specifies the `autorater_model` by providing the resource name of the deployed Selene endpoint.\n",
        "\n",
        "Running the `.evaluate()` method initiates the job. This process iterates through the dataset, formats the evaluation prompt for each row using the metric template, sends it to the Selene autorater endpoint, parses the score and reasoning, and returns the results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RXtyXj3UAr8l"
      },
      "outputs": [],
      "source": [
        "eval_result = EvalTask(\n",
        "    dataset=human_rated_dataset,\n",
        "    metrics=[\n",
        "        completeness,\n",
        "    ],\n",
        "    experiment=EXPERIMENT_NAME,\n",
        "    autorater_config=AutoraterConfig(\n",
        "        autorater_model=deployed_judge_model.resource_name\n",
        "    ),\n",
        "    output_uri_prefix=BUCKET_URI + \"/evaluation_results\",\n",
        ").evaluate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qr1EIwcsQH9i"
      },
      "source": [
        "### Evaluate the autorater\n",
        "\n",
        "After the autorater has generated its scores, you may what to  assess how well those scores align with the pre-existing human ratings for the same metric ('completeness').\n",
        "\n",
        "The `evaluate_autorater` utility function is used for this purpose. It takes the evaluation results table (containing both human and autorater scores) and the metric definition as input. The function calculates various comparison statistics, and a confusion matrix, which directly compares the distribution of scores between the human rater and the autorater model. These statistics help quantify the reliability of the autorater."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tePYdF5UQT_G"
      },
      "outputs": [],
      "source": [
        "evaluate_autorater_result = evaluate_autorater(\n",
        "    evaluate_autorater_input=eval_result.metrics_table, eval_metrics=[completeness]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ODNna5rgaTs4"
      },
      "outputs": [],
      "source": [
        "df_data = eval_result.metrics_table[[\"completeness/human_rating\", \"completeness/score\"]]\n",
        "metrics = evaluate_autorater_result.eval_result"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DemUGUe_FleQ"
      },
      "source": [
        "### Visualize Judge model alignement\n",
        "\n",
        "Use some visualization to represent the comparison between the autorater's scores and the human ratings using the Plotly library. Three types of plots are generated:\n",
        "\n",
        "1.  **Distribution Bar Charts:** Side-by-side histograms showing the frequency of each score (1-5) given by humans versus the autorater, allowing for a quick comparison of overall scoring tendencies.\n",
        "2.  **Confusion Matrix Heatmap:** A heatmap visualizing the agreement and disagreement between human ratings (y-axis) and autorater scores (x-axis). Strong agreement appears along the diagonal.\n",
        "3.  **Jitter Scatter Plot:** Each point represents an evaluated item, plotted by its autorater score (x) and human rating (y). A small amount of \"jitter\" (random noise) is added to prevent points from overlapping perfectly, making density clearer. An \"Ideal Alignment\" line (y=x) is included for reference; points close to this line indicate good agreement.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5lTOjimWtj5g"
      },
      "outputs": [],
      "source": [
        "run_visual_analysis(df_data=df_data, metrics=metrics)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2a4e033321ad"
      },
      "source": [
        "## Cleaning up"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K7z_Oe8VuzDA"
      },
      "outputs": [],
      "source": [
        "delete_endpoint = False\n",
        "delete_experiments = False\n",
        "\n",
        "if delete_endpoint:\n",
        "    deployed_judge_model.delete(force=True)\n",
        "\n",
        "if delete_experiments:\n",
        "    experiments = aiplatform.Experiment.list()\n",
        "    for experiment in experiments:\n",
        "        if experiment.name == EXPERIMENT_NAME:\n",
        "            experiment.delete()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "vertex_ai_tgi_evaluate_llm_with_open_judge.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
