{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-PKa6W4wdPWr"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idM_aIPheQDG"
      },
      "source": [
        "# Stage 2: Building MVP: - 05 Evaluation with Vertex AI\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fworkshops%2Frag-ops%2F2.5_mvp_evaluation_vertexai_eval.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/workshops/rag-ops/2.5_mvp_evaluation_vertexai_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o0oOZgLKfhCi"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook is the fifth in a series designed to guide you through building a Minimum Viable Product (MVP) for a Multimodal Retrieval Augmented Generation (RAG) system using the Vertex Gemini API.\n",
        "\n",
        "This notebook dives deeper into evaluating RAG system performance by introducing Vertex AI Eval service, a powerful tool for assessing the quality of generated answers. Building upon the previous notebook's hands-on approach to evaluation, we now explore a more streamlined and scalable method using Vertex AI's dedicated evaluation capabilities.\n",
        "\n",
        "**Here's what you'll achieve:**\n",
        "\n",
        "* **Harness the Power of Vertex AI Eval Service:** Learn to effectively utilize Vertex AI Eval service to evaluate answers generated by your RAG system. This involves understanding its features, configuring evaluation jobs, and interpreting the results.  You can find more information about Vertex AI Eval service in the [official documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview).\n",
        "* **Explore Predefined and Custom Metrics:**  Explore the range of predefined metrics offered by Vertex AI Eval service and learn how to create custom metrics tailored to your specific evaluation needs.  The [documentation on defining evaluation metrics](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval) provides a comprehensive guide.\n",
        "* **Streamline Your Evaluation Workflow:** Experience a more efficient and scalable evaluation process compared to manual implementation. Vertex AI Eval service automates many underlying tasks, allowing you to focus on analyzing results and improving your system.\n",
        "* **Gain Deeper Insights with Visualizations:**  Utilize the visualization capabilities of Vertex AI Eval service to gain a comprehensive understanding of your RAG system's performance. Explore various visualizations to identify specific areas for improvement, such as factual accuracy, coherence, and relevance.\n",
        "* **Compare and Contrast with Detailed Explanations:**  Continue the analysis of Gemini 2.0 and Gemini 2.0 models by evaluating their performance using Vertex AI Eval service.  Leverage the service's ability to provide detailed explanations for individual instances to understand the strengths and weaknesses of each model.\n",
        "\n",
        "This notebook offers a practical introduction to Vertex AI Eval service and its application in evaluating RAG systems. By leveraging this powerful tool, you can streamline your evaluation workflow, gain deeper insights into your system's performance, and make data-driven improvements to build a more robust and reliable RAG MVP."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6KGP8kNhfklW"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HGFDxQ7_flui"
      },
      "source": [
        "### Install Vertex AI SDK for Python\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "YcWmeELUeTPq"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SYYzbNlmfvKJ"
      },
      "source": [
        "### Restart runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "icVKaoLAfw6c"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oZoekSN8fy2E"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E4rlvfF1f1RA"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the cell below to authenticate your environment.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "XTaUAqKLf3N8"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iQJQgZKXf6Mx"
      },
      "source": [
        "### Set Google Cloud project information, GCS Bucket and initialize Vertex AI SDK\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "FqXU2Ptvf5LA"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import sys\n",
        "\n",
        "from google.cloud import storage\n",
        "import vertexai\n",
        "\n",
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"\n",
        "BUCKET_NAME = \"mlops-for-genai\"\n",
        "EXPERIMENT = \"rag-eval-01\"\n",
        "\n",
        "if PROJECT_ID == \"[your-project-id]\":\n",
        "    PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
        "\n",
        "if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\" or PROJECT_ID == \"None\":\n",
        "    raise ValueError(\"Please set your PROJECT_ID\")\n",
        "\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
        "\n",
        "# Initialize cloud storage\n",
        "storage_client = storage.Client(project=PROJECT_ID)\n",
        "bucket = storage_client.bucket(BUCKET_NAME)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "6AbGiKORhxQ7"
      },
      "outputs": [],
      "source": [
        "# # Variables for data location. Do not change.\n",
        "\n",
        "PRODUCTION_DATA = \"multimodal-finanace-qa/data/unstructured/production/\"\n",
        "PICKLE_FILE_NAME = \"training_data_results.pkl\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s9-h_WOcgAKX"
      },
      "source": [
        "### Import libraries\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "OaW7NsbHgAoo"
      },
      "outputs": [],
      "source": [
        "import inspect\n",
        "import logging\n",
        "import pickle\n",
        "import warnings\n",
        "\n",
        "from IPython.display import HTML, Markdown, display\n",
        "from google.cloud import storage\n",
        "\n",
        "# Library\n",
        "import pandas as pd\n",
        "import plotly.graph_objects as go\n",
        "from vertexai.evaluation import EvalTask, MetricPromptTemplateExamples, PointwiseMetric\n",
        "from vertexai.generative_models import GenerativeModel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "j3h2cbsow9as"
      },
      "outputs": [],
      "source": [
        "logging.getLogger(\"urllib3.connectionpool\").setLevel(logging.ERROR)\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A9hij3nhgDz8"
      },
      "source": [
        "### Load the Gemini 2.0 models\n",
        "\n",
        "To learn more about all [Gemini API models on Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models).\n",
        "\n",
        "The Gemini model family has several model versions. You will start by using Gemini 2.0. Gemini 2.0 is a more lightweight, fast, and cost-efficient model. This makes it a great option for prototyping.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "7xvFkagTgHMc"
      },
      "outputs": [],
      "source": [
        "MODEL_ID_FLASH = \"gemini-2.0-flash\"  # @param {type:\"string\"}\n",
        "MODEL_ID_PRO = \"gemini-2.0-flash\"  # @param {type:\"string\"}\n",
        "\n",
        "\n",
        "gemini_15_flash = GenerativeModel(MODEL_ID_FLASH)\n",
        "gemini_15_pro = GenerativeModel(MODEL_ID_PRO)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "cellView": "form",
        "id": "oS1ar31xYmkR"
      },
      "outputs": [],
      "source": [
        "# @title Helper Functions\n",
        "\n",
        "\n",
        "def get_load_dataframes_from_gcs():\n",
        "    gcs_path = \"multimodal-finanace-qa/data/embeddings/index_db.pkl\"\n",
        "    # print(\"GCS PAth: \", gcs_path)\n",
        "    blob = bucket.blob(gcs_path)\n",
        "\n",
        "    # Download the pickle file from GCS\n",
        "    blob.download_to_filename(f\"{PICKLE_FILE_NAME}\")\n",
        "\n",
        "    # Load the pickle file into a list of dataframes\n",
        "    with open(f\"{PICKLE_FILE_NAME}\", \"rb\") as f:\n",
        "        dataframes = pickle.load(f)\n",
        "\n",
        "    # Assign the dataframes to variables\n",
        "    (\n",
        "        index_db_final,\n",
        "        extracted_text_chunk_df,\n",
        "        video_metadata_chunk_df,\n",
        "        audio_metadata_chunk_df,\n",
        "    ) = dataframes\n",
        "\n",
        "    return (\n",
        "        index_db_final,\n",
        "        extracted_text_chunk_df,\n",
        "        video_metadata_chunk_df,\n",
        "        audio_metadata_chunk_df,\n",
        "    )\n",
        "\n",
        "\n",
        "def get_load_training_dataframes_from_gcs():\n",
        "    gcs_path = \"multimodal-finanace-qa/data/structured/\" + PICKLE_FILE_NAME\n",
        "    # print(\"GCS PAth: \", gcs_path)\n",
        "    blob = bucket.blob(gcs_path)\n",
        "\n",
        "    # Download the pickle file from GCS\n",
        "    blob.download_to_filename(f\"{PICKLE_FILE_NAME}\")\n",
        "\n",
        "    # Load the pickle file into a list of dataframes\n",
        "    with open(f\"{PICKLE_FILE_NAME}\", \"rb\") as f:\n",
        "        dataframes = pickle.load(f)\n",
        "\n",
        "    # Assign the dataframes to variables\n",
        "    training_data_flash, training_data_pro = dataframes\n",
        "\n",
        "    return training_data_flash, training_data_pro"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KKT4gg2ybwaR"
      },
      "source": [
        "![](https://storage.googleapis.com/mlops-for-genai/multimodal-finanace-qa/img/rag_eval_flow.png)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "Jk5387FCjlAv"
      },
      "outputs": [],
      "source": [
        "# Get the data that has been extracted in the previous step: IndexDB.\n",
        "# Make sure that you have ran the previous notebook: stage_2_mvp_chunk_embeddings.ipynb\n",
        "\n",
        "\n",
        "(\n",
        "    index_db_final,\n",
        "    extracted_text_chunk_df,\n",
        "    video_metadata_chunk_df,\n",
        "    audio_metadata_chunk_df,\n",
        ") = get_load_dataframes_from_gcs()\n",
        "training_data_flash, training_data_pro = get_load_training_dataframes_from_gcs()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "isgFDZBZyZ_1"
      },
      "outputs": [],
      "source": [
        "index_db_final.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "_8oBQwejBjqV"
      },
      "outputs": [],
      "source": [
        "training_data_flash.head(2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-aA38s_JBlNB"
      },
      "outputs": [],
      "source": [
        "training_data_pro.head(2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dtLLgOhH5yv-"
      },
      "outputs": [],
      "source": [
        "training_data_pro.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltfT0cm-lUEc"
      },
      "source": [
        "### Evaluations"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "cellView": "form",
        "id": "wTarzEqLw7Nd"
      },
      "outputs": [],
      "source": [
        "# @title Vertex AI Eval Helper Functions\n",
        "\n",
        "\n",
        "def print_doc(function):\n",
        "    print(f\"{function.__name__}:\\n{inspect.getdoc(function)}\\n\")\n",
        "\n",
        "\n",
        "def display_eval_report(eval_result, metrics=None):\n",
        "    \"\"\"Display the evaluation results.\"\"\"\n",
        "\n",
        "    title, summary_metrics, report_df = eval_result\n",
        "    metrics_df = pd.DataFrame.from_dict(summary_metrics, orient=\"index\").T\n",
        "    if metrics:\n",
        "        metrics_df = metrics_df.filter(\n",
        "            [\n",
        "                metric\n",
        "                for metric in metrics_df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "        report_df = report_df.filter(\n",
        "            [\n",
        "                metric\n",
        "                for metric in report_df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "\n",
        "    # Display the title with Markdown for emphasis\n",
        "    display(Markdown(f\"## {title}\"))\n",
        "\n",
        "    # Display the metrics DataFrame\n",
        "    display(Markdown(\"### Summary Metrics\"))\n",
        "    display(metrics_df)\n",
        "\n",
        "    # Display the detailed report DataFrame\n",
        "    display(Markdown(\"### Report Metrics\"))\n",
        "    display(report_df)\n",
        "\n",
        "\n",
        "def display_explanations(df, metrics=None, n=1):\n",
        "    style = \"white-space: pre-wrap; width: 800px; overflow-x: auto;\"\n",
        "    df = df.sample(n=n)\n",
        "    if metrics:\n",
        "        df = df.filter(\n",
        "            [\"instruction\", \"context\", \"reference\", \"completed_prompt\", \"response\"]\n",
        "            + [\n",
        "                metric\n",
        "                for metric in df.columns\n",
        "                if any(selected_metric in metric for selected_metric in metrics)\n",
        "            ]\n",
        "        )\n",
        "\n",
        "    for index, row in df.iterrows():\n",
        "        for col in df.columns:\n",
        "            display(HTML(f\"<h2>{col}:</h2> <div style='{style}'>{row[col]}</div>\"))\n",
        "        display(HTML(\"<hr>\"))\n",
        "\n",
        "\n",
        "def plot_radar_plot(eval_results, max_score=5, metrics=None):\n",
        "    fig = go.Figure()\n",
        "\n",
        "    for eval_result in eval_results:\n",
        "        title, summary_metrics, report_df = eval_result\n",
        "\n",
        "        if metrics:\n",
        "            summary_metrics = {\n",
        "                k: summary_metrics[k]\n",
        "                for k, v in summary_metrics.items()\n",
        "                if any(selected_metric in k for selected_metric in metrics)\n",
        "            }\n",
        "\n",
        "        fig.add_trace(\n",
        "            go.Scatterpolar(\n",
        "                r=list(summary_metrics.values()),\n",
        "                theta=list(summary_metrics.keys()),\n",
        "                fill=\"toself\",\n",
        "                name=title,\n",
        "            )\n",
        "        )\n",
        "\n",
        "    fig.update_layout(\n",
        "        polar=dict(radialaxis=dict(visible=True, range=[0, max_score])), showlegend=True\n",
        "    )\n",
        "\n",
        "    fig.show()\n",
        "\n",
        "\n",
        "def plot_bar_plot(eval_results, metrics=None):\n",
        "    fig = go.Figure()\n",
        "    data = []\n",
        "\n",
        "    for eval_result in eval_results:\n",
        "        title, summary_metrics, _ = eval_result\n",
        "        if metrics:\n",
        "            summary_metrics = {\n",
        "                k: summary_metrics[k]\n",
        "                for k, v in summary_metrics.items()\n",
        "                if any(selected_metric in k for selected_metric in metrics)\n",
        "            }\n",
        "\n",
        "        data.append(\n",
        "            go.Bar(\n",
        "                x=list(summary_metrics.keys()),\n",
        "                y=list(summary_metrics.values()),\n",
        "                name=title,\n",
        "            )\n",
        "        )\n",
        "\n",
        "    fig = go.Figure(data=data)\n",
        "\n",
        "    # Change the bar mode\n",
        "    fig.update_layout(barmode=\"group\")\n",
        "    fig.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EZKO8V7YxQJp"
      },
      "source": [
        "### Prepare your dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R-qTlWsNxORQ"
      },
      "source": [
        "To evaluate the RAG generated answers, the evaluation dataset is required to contain the following fields:\n",
        "\n",
        "* Prompt: The user supplied prompt consisting of the User Question and the RAG Retrieved Context\n",
        "* Response: The RAG Generated Answer\n",
        "* Reference: The Golden Answer groundtruth to compare model response to"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U6WvYZ2yxkn3"
      },
      "source": [
        "### rag_a: Gemini 2.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "PMZ8TwSj4Sjg"
      },
      "outputs": [],
      "source": [
        "def get_citation(row):\n",
        "    final_ref = []\n",
        "    for each_cit in row[:20]:\n",
        "        final_ref.append(each_cit[\"content\"])\n",
        "    return \"\\n\".join(final_ref)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "DmLO5nR_xGqC"
      },
      "outputs": [],
      "source": [
        "questions = training_data_pro[\"question\"].tolist()\n",
        "\n",
        "retrieved_contexts = training_data_pro[\"citation\"].apply(get_citation).tolist()\n",
        "\n",
        "generated_answers_by_rag_a = training_data_pro[\"gen_answer\"].tolist()\n",
        "\n",
        "\n",
        "golden_answers = training_data_pro[\"answer\"].tolist()\n",
        "\n",
        "referenced_eval_dataset_rag_a = pd.DataFrame(\n",
        "    {\n",
        "        \"prompt\": [\n",
        "            \"Answer the question: \" + question + \" Context: \" + item\n",
        "            for question, item in zip(questions, retrieved_contexts)\n",
        "        ],\n",
        "        \"response\": generated_answers_by_rag_a,\n",
        "        \"reference\": golden_answers,\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5CKM9p6xq-J"
      },
      "source": [
        "### rag_b: Gemini 2.0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "BcTZr7X7xWx0"
      },
      "outputs": [],
      "source": [
        "questions = training_data_flash[\"question\"].tolist()\n",
        "\n",
        "retrieved_contexts = training_data_flash[\"citation\"].apply(get_citation).tolist()\n",
        "\n",
        "\n",
        "generated_answers_by_rag_b = training_data_flash[\"gen_answer\"].tolist()\n",
        "\n",
        "golden_answers = training_data_flash[\"answer\"].tolist()\n",
        "\n",
        "\n",
        "referenced_eval_dataset_rag_b = pd.DataFrame(\n",
        "    {\n",
        "        \"prompt\": [\n",
        "            \"Answer the question: \" + question + \" Context: \" + item\n",
        "            for question, item in zip(questions, retrieved_contexts)\n",
        "        ],\n",
        "        \"response\": generated_answers_by_rag_b,\n",
        "        \"reference\": golden_answers,\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fhVpbNGX7eZU"
      },
      "source": [
        "### Select and create metrics\n",
        "\n",
        "\n",
        "You can run evaluation for just one metric, or a combination of metrics. For this example, we select a few RAG-related predefined metrics, and create a few of our own custom metrics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dT9esdgi7XHA"
      },
      "source": [
        "#### Explore predefined metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "fsXMoHA55Tm0"
      },
      "outputs": [],
      "source": [
        "# See all the available metric examples\n",
        "MetricPromptTemplateExamples.list_example_metric_names()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "hJf5uHci7a1W"
      },
      "outputs": [],
      "source": [
        "# See the prompt example for one of the pointwise metrics\n",
        "print(MetricPromptTemplateExamples.get_prompt_template(\"question_answering_quality\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_qOzc0k6Jrr"
      },
      "source": [
        "#### Create custom metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "7s60Ne5t59jr"
      },
      "outputs": [],
      "source": [
        "relevance_prompt_template = \"\"\"\n",
        "You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.\n",
        "\n",
        "You will be assessing relevance, which measures the ability to respond with relevant information when given a prompt.\n",
        "\n",
        "You will assign the writing response a score from 5, 4, 3, 2, 1, following the rating rubric and evaluation steps.\n",
        "\n",
        "## Criteria\n",
        "Relevance: The response should be relevant to the instruction and directly address the instruction.\n",
        "\n",
        "## Rating Rubric\n",
        "5 (completely relevant): Response is entirely relevant to the instruction and provides clearly defined information that addresses the instruction's core needs directly.\n",
        "4 (mostly relevant): Response is mostly relevant to the instruction and addresses the instruction mostly directly.\n",
        "3 (somewhat relevant): Response is somewhat relevant to the instruction and may address the instruction indirectly, but could be more relevant and more direct.\n",
        "2 (somewhat irrelevant): Response is minimally relevant to the instruction and does not address the instruction directly.\n",
        "1 (irrelevant): Response is completely irrelevant to the instruction.\n",
        "\n",
        "## Evaluation Steps\n",
        "STEP 1: Assess relevance: is response relevant to the instruction and directly address the instruction?\n",
        "STEP 2: Score based on the criteria and rubrics.\n",
        "\n",
        "Give step by step explanations for your scoring, and only choose scores from 5, 4, 3, 2, 1.\n",
        "\n",
        "\n",
        "# User Inputs and AI-generated Response\n",
        "## User Inputs\n",
        "### Prompt\n",
        "{prompt}\n",
        "\n",
        "## AI-generated Response\n",
        "{response}\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "aGSienWj5_o4"
      },
      "outputs": [],
      "source": [
        "helpfulness_prompt_template = \"\"\"\n",
        "You are a professional writing evaluator. Your job is to score writing responses according to pre-defined evaluation criteria.\n",
        "\n",
        "You will be assessing helpfulness, which measures the ability to provide important details when answering a prompt.\n",
        "\n",
        "You will assign the writing response a score from 5, 4, 3, 2, 1, following the rating rubric and evaluation steps.\n",
        "\n",
        "## Criteria\n",
        "Helpfulness: The response is comprehensive with well-defined key details. The user would feel very satisfied with the content in a good response.\n",
        "\n",
        "## Rating Rubric\n",
        "5 (completely helpful): Response is useful and very comprehensive with well-defined key details to address the needs in the instruction and usually beyond what explicitly asked. The user would feel very satisfied with the content in the response.\n",
        "4 (mostly helpful): Response is very relevant to the instruction, providing clearly defined information that addresses the instruction's core needs.  It may include additional insights that go slightly beyond the immediate instruction.  The user would feel quite satisfied with the content in the response.\n",
        "3 (somewhat helpful): Response is relevant to the instruction and provides some useful content, but could be more relevant, well-defined, comprehensive, and/or detailed. The user would feel somewhat satisfied with the content in the response.\n",
        "2 (somewhat unhelpful): Response is minimally relevant to the instruction and may provide some vaguely useful information, but it lacks clarity and detail. It might contain minor inaccuracies. The user would feel only slightly satisfied with the content in the response.\n",
        "1 (unhelpful): Response is useless/irrelevant, contains inaccurate/deceptive/misleading information, and/or contains harmful/offensive content. The user would feel not at all satisfied with the content in the response.\n",
        "\n",
        "## Evaluation Steps\n",
        "STEP 1: Assess comprehensiveness: does the response provide specific, comprehensive, and clearly defined information for the user needs expressed in the instruction?\n",
        "STEP 2: Assess relevance: When appropriate for the instruction, does the response exceed the instruction by providing relevant details and related information to contextualize content and help the user better understand the response.\n",
        "STEP 3: Assess accuracy: Is the response free of inaccurate, deceptive, or misleading information?\n",
        "STEP 4: Assess safety: Is the response free of harmful or offensive content?\n",
        "\n",
        "Give step by step explanations for your scoring, and only choose scores from 5, 4, 3, 2, 1.\n",
        "\n",
        "\n",
        "# User Inputs and AI-generated Response\n",
        "## User Inputs\n",
        "### Prompt\n",
        "{prompt}\n",
        "\n",
        "## AI-generated Response\n",
        "{response}\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "1rJVUQvB5ZXs"
      },
      "outputs": [],
      "source": [
        "relevance = PointwiseMetric(\n",
        "    metric=\"relevance\",\n",
        "    metric_prompt_template=relevance_prompt_template,\n",
        ")\n",
        "\n",
        "helpfulness = PointwiseMetric(\n",
        "    metric=\"helpfulness\",\n",
        "    metric_prompt_template=helpfulness_prompt_template,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHsuA2UJ6Cku"
      },
      "source": [
        "### Run evaluation with your dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "DomvnRUS6DKb"
      },
      "outputs": [],
      "source": [
        "rag_eval_task_rag_a = EvalTask(\n",
        "    dataset=referenced_eval_dataset_rag_a,\n",
        "    metrics=[\n",
        "        \"question_answering_quality\",\n",
        "        relevance,\n",
        "        helpfulness,\n",
        "        \"groundedness\",\n",
        "        \"safety\",\n",
        "        \"instruction_following\",\n",
        "    ],\n",
        "    experiment=EXPERIMENT,\n",
        ")\n",
        "\n",
        "rag_eval_task_rag_b = EvalTask(\n",
        "    dataset=referenced_eval_dataset_rag_b,\n",
        "    metrics=[\n",
        "        \"question_answering_quality\",\n",
        "        relevance,\n",
        "        helpfulness,\n",
        "        \"groundedness\",\n",
        "        \"safety\",\n",
        "        \"instruction_following\",\n",
        "    ],\n",
        "    experiment=EXPERIMENT,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "I54Zd8Zm6EyO"
      },
      "outputs": [],
      "source": [
        "result_rag_a = rag_eval_task_rag_a.evaluate()\n",
        "result_rag_b = rag_eval_task_rag_b.evaluate()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No16Glcy60g2"
      },
      "source": [
        "### Display evaluation results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lBsiKlpl605R"
      },
      "source": [
        "#### View summary results\n",
        "\n",
        "If you want to have an overall view of all the metrics from individual model's evaluation result in one table, you can use the `display_eval_report()` helper function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "XUg6wKms64WR"
      },
      "outputs": [],
      "source": [
        "display_eval_report(\n",
        "    (\n",
        "        \"Model A Eval Result\",\n",
        "        result_rag_a.summary_metrics,\n",
        "        result_rag_a.metrics_table,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "sO7BhPdd66G2"
      },
      "outputs": [],
      "source": [
        "display_eval_report(\n",
        "    (\n",
        "        \"Model B Eval Result\",\n",
        "        result_rag_b.summary_metrics,\n",
        "        result_rag_b.metrics_table,\n",
        "    )\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xxVInnMo68ow"
      },
      "source": [
        "#### Visualize evaluation results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "QJ-o4w8q68-V"
      },
      "outputs": [],
      "source": [
        "eval_results = []\n",
        "eval_results.append(\n",
        "    (\"Model A\", result_rag_a.summary_metrics, result_rag_a.metrics_table)\n",
        ")\n",
        "eval_results.append(\n",
        "    (\"Model B\", result_rag_b.summary_metrics, result_rag_b.metrics_table)\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "pVc0Z-nE6_rH"
      },
      "outputs": [],
      "source": [
        "plot_radar_plot(\n",
        "    eval_results,\n",
        "    metrics=[\n",
        "        f\"{metric}/mean\"\n",
        "        # Edit your list of metrics here if you used other metrics in evaluation.\n",
        "        for metric in [\n",
        "            \"question_answering_quality\",\n",
        "            \"safety\",\n",
        "            \"groundedness\",\n",
        "            \"instruction_following\",\n",
        "            \"relevance\",\n",
        "            \"helpfulness\",\n",
        "        ]\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "FmnenV7-7BVS"
      },
      "outputs": [],
      "source": [
        "plot_bar_plot(\n",
        "    eval_results,\n",
        "    metrics=[\n",
        "        f\"{metric}/mean\"\n",
        "        for metric in [\n",
        "            \"question_answering_quality\",\n",
        "            \"safety\",\n",
        "            \"groundedness\",\n",
        "            \"instruction_following\",\n",
        "            \"relevance\",\n",
        "            \"helpfulness\",\n",
        "        ]\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YK-oWcJI7DNL"
      },
      "source": [
        "#### View detailed explanation for an individual instance\n",
        "\n",
        "If you need to delve into the individual result's detailed explanations on why a score is assigned and how confident the model is for each model-based metric, you can use the `display_explanations()` helper function. For example, you can set `n=2` to display explanation of the 2nd instance result as follows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "id": "cq8xZHwo7E1Q"
      },
      "outputs": [],
      "source": [
        "display_explanations(result_rag_a.metrics_table, n=2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LGLhRyHi7GsV"
      },
      "source": [
        "You can also focus on one or a few metrics as follows."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "sVLWtLQ27Ibh"
      },
      "outputs": [],
      "source": [
        "display_explanations(result_rag_b.metrics_table, metrics=[\"groundedness\"])"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "2.5_mvp_evaluation_vertexai_eval.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
