{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ijGzTHJJUCPY"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VEqbX8OhE8y9"
      },
      "source": [
        "# Enhancing quality and explainability with Vertex AI Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4d450b43efab"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fevaluation%2Fevaltask_approach%2Fenhancing_quality_and_explainability_with_eval.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/evaltask_approach/enhancing_quality_and_explainability_with_eval.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ce84bd67392c"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "| Author(s) | [Anant Nawalgaria](https://github.com/anantnawal) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CkHPv2myT2cx"
      },
      "source": [
        "## Overview\n",
        "\n",
        "### Vertex Gen AI Evaluation API\n",
        "\n",
        "The [Vertex Gen AI Evaluation Service](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) which can be accessed both through its SDK and web API interfaces, lets you evaluate your large language models (LLMs), both pointwise and pairwise, across several metrics.\n",
        "\n",
        "It is primarily used ad-hoc in the initial experimental phase for evaluating which set of prompts and models work well for a use case. However, as described in detail in the corresponding blog, this notebook will show some sample code on dummy data of how you can use \n",
        "Evaluation to enhance the quality of the response generated by the LLMs by combining the pairwise and pointwise capabilities of Gen AI Evaluation elegantly at the time of generation. It would also then return a human readable explanation to help understand the quality evaluation of the response. Note that although this notebook only demonstrates this workflow on text, it can be extended to any modality once an evaluation mechanism is available for that modality.\n",
        "\n",
        "For more information about generative AI on Vertex AI please see [Generative AI on Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview) documentation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DrkcqHrrwMAo"
      },
      "source": [
        "### Objectives\n",
        "\n",
        "In this tutorial, you will learn how to combine the Gemini API in Vertex AI  with the Gen AI Eval API service for Python to improve generation quality & explainability of the responses.\n",
        "You will complete the following tasks:\n",
        "\n",
        "- Install the Vertex AI SDK for Python\n",
        "- Use the Gemini API in Vertex AI to interact with each model\n",
        "  - Gemini 2.0 (`gemini-2.0-flash`) model:\n",
        "    - Generate multiple responses for a given instruction and context\n",
        "    - Use the pairwise and pointwise capabilities of eval to select the best response and also return a human readable explanation for it. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C9nEPojogw-g"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "- Vertex AI\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r11Gu7qNgx1p"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install the required libraries for Python"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user --quiet google-cloud-aiplatform[evaluation]\n",
        "%pip install --upgrade --user bigframes -q\n",
        "%pip install --quiet --upgrade nest_asyncio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j7UyNVSiyQ96"
      },
      "source": [
        "### Restart current runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, it is recommended to restart the runtime. Run the following cell to restart the current kernel.\n",
        "\n",
        "The restart process might take a minute or so."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YmY9HVVGSBW5"
      },
      "outputs": [],
      "source": [
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5b21ea7cdbf7"
      },
      "source": [
        "#### Set your project ID and region"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "14cc8984ed02"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"\n",
        "LOCATION = \"us-central1\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ea53274ede4d"
      },
      "source": [
        "After the restart is complete, continue to the next step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EXQZrM5hQeKb"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ Wait for the kernel to finish restarting before you continue. ⚠️</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the following cell to authenticate your environment. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# Additional authentication is required for Google Colab\n",
        "if \"google.colab\" in sys.modules:\n",
        "    # Authenticate user to Google Cloud\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Define Google Cloud project information (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, specify the Google Cloud project information to use. In the following cell, you specify your project information, import the Vertex AI package, and initialize the package. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "if \"google.colab\" in sys.modules:\n",
        "    # Define project information\n",
        "    # Initialize Vertex AI\n",
        "    import vertexai\n",
        "\n",
        "    vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jXHfaVS66_01"
      },
      "source": [
        "### Import libraries & initialize project variables"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "lslYAvw37JGQ"
      },
      "outputs": [],
      "source": [
        "import functools\n",
        "import uuid\n",
        "from functools import partial\n",
        "\n",
        "import nest_asyncio\n",
        "import pandas as pd\n",
        "from google.cloud import aiplatform\n",
        "from vertexai.evaluation import EvalTask, MetricPromptTemplateExamples\n",
        "from vertexai.generative_models import GenerationConfig, GenerativeModel\n",
        "\n",
        "nest_asyncio.apply()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39ce45a03542"
      },
      "source": [
        "## Defining functions for ranking using evaluations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4437b7608c8e"
      },
      "source": [
        "This section defines the various helper functions to perform pairwise and pointwise evaluations, as well as the logic to combine them \n",
        "to select the best response and return associated quality metrics and explanation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "afc4054f1f70"
      },
      "source": [
        "This function simplifies AutoSXS comparisons between pairs of responses. It uses the Gen AI Evaluation Service API in Vertex AI and works well with Python's <code>max()</code> or <code>sorted()</code> functions. This lets you easily find the best response or rank a list of responses using pairwise comparisons. For other tasks, like summarization SxS, you can find a full list of metrics on the website mentioned below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ac10531efd0c"
      },
      "outputs": [],
      "source": [
        "experiment_name = \"qa-quality\"\n",
        "\n",
        "\n",
        "def pairwise_greater(\n",
        "    instructions: list,\n",
        "    context: str,\n",
        "    project_id: str,\n",
        "    location: str,\n",
        "    experiment_name: str,\n",
        "    baseline: str,\n",
        "    candidate: str,\n",
        ") -> tuple:\n",
        "    \"\"\"Takes Instructions, Context and two different responses.\n",
        "    Returns the response which best matches the instructions/Context for the given\n",
        "    quality metric ( in this case question answering).\n",
        "    More details on the web API and different quality metrics which this function\n",
        "    can be extended to can be found on\n",
        "    https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/evaluation\n",
        "    \"\"\"\n",
        "    eval_dataset = pd.DataFrame(\n",
        "        {\n",
        "            \"instruction\": [instructions],\n",
        "            \"context\": [context],\n",
        "            \"response\": [candidate],\n",
        "            \"baseline_model_response\": [baseline],\n",
        "        }\n",
        "    )\n",
        "\n",
        "    eval_task = EvalTask(\n",
        "        dataset=eval_dataset,\n",
        "        metrics=[\n",
        "            MetricPromptTemplateExamples.Pairwise.QUESTION_ANSWERING_QUALITY,\n",
        "        ],\n",
        "        experiment=experiment_name,\n",
        "    )\n",
        "    results = eval_task.evaluate(\n",
        "        prompt_template=\"{instruction} \\n {context}\",\n",
        "        experiment_run_name=\"gemini-qa-pairwise-\" + str(uuid.uuid4()),\n",
        "    )\n",
        "    result = results.metrics_table[\n",
        "        [\n",
        "            \"pairwise_question_answering_quality/pairwise_choice\",\n",
        "            \"pairwise_question_answering_quality/explanation\",\n",
        "        ]\n",
        "    ].to_dict(\"records\")[0]\n",
        "    choice = (\n",
        "        baseline\n",
        "        if result[\"pairwise_question_answering_quality/pairwise_choice\"] == \"BASELINE\"\n",
        "        else candidate\n",
        "    )\n",
        "    return (choice, result[\"pairwise_question_answering_quality/explanation\"])\n",
        "\n",
        "\n",
        "def greater(cmp: callable, a: str, b: str) -> int:\n",
        "    \"\"\"A comparison function which takes the comparison function, and two variables as input\n",
        "    and returns the one which is greater according to the logic defined inside the cmp function.\n",
        "    \"\"\"\n",
        "    choice, explanation = cmp(a, b)\n",
        "\n",
        "    if choice == a:\n",
        "        return 1\n",
        "    return -1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "602441b87ef7"
      },
      "source": [
        "The below function performs the pointwise evaluation of the provided set of responses, with respect to the provided metric, instruction and context."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6863af9c235f"
      },
      "outputs": [],
      "source": [
        "def pointwise_eval(\n",
        "    instruction: str,\n",
        "    context: str,\n",
        "    responses: list[str],\n",
        "    eval_metrics: list[object] = [\n",
        "        MetricPromptTemplateExamples.Pointwise.QUESTION_ANSWERING_QUALITY,\n",
        "        MetricPromptTemplateExamples.Pointwise.GROUNDEDNESS,\n",
        "    ],\n",
        "    experiment_name: str = experiment_name,\n",
        ") -> object:\n",
        "    \"\"\"Takes the instruction, context and a variable number of corresponding generated responses, and returns the pointwise evaluation metrics\n",
        "    for each of the provided metrics. For this example the metrics are Q & A related, however the full list can be found on the website:\n",
        "    https://cloud.google.com/vertex-ai/generative-ai/docs/models/online-pipeline-services\n",
        "    \"\"\"\n",
        "    instructions = [instruction] * len(responses)\n",
        "\n",
        "    contexts = [context] * len(responses)\n",
        "\n",
        "    eval_dataset = pd.DataFrame(\n",
        "        {\n",
        "            \"instruction\": instructions,\n",
        "            \"context\": contexts,\n",
        "            \"response\": responses,\n",
        "        }\n",
        "    )\n",
        "\n",
        "    eval_task = EvalTask(\n",
        "        dataset=eval_dataset, metrics=eval_metrics, experiment=experiment_name\n",
        "    )\n",
        "    results = eval_task.evaluate(\n",
        "        prompt_template=\"{instruction} \\n {context}\",\n",
        "        experiment_run_name=\"gemini-qa-pointwise-\" + str(uuid.uuid4()),\n",
        "    )\n",
        "    (results.metrics_table.columns)\n",
        "    return results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "971acda8dd12"
      },
      "source": [
        "This function integrates pairwise and pointwise logic to enhance response selection and evaluation. Here's the process:\n",
        "\n",
        "**Current Workflow**:\n",
        "\n",
        "1. Pairwise Comparison: Compares responses in pairs to identify the best one based on user-defined metrics.\n",
        "2. Pointwise Evaluation: Assesses the quality of the chosen response, providing human-readable explanations to build trust.\n",
        "\n",
        "**Alternative Workflow**:\n",
        "\n",
        "1. Pointwise Evaluation: Evaluate each response individually based on the Pointwise scores or likelihood/logprobs of the response, filtering out those that don't meet specific quality criteria or selecting the top K responses.\n",
        "2. Pairwise Comparison: Ranks the remaining high-quality responses using pairwise methods to determine the best one(s).\n",
        "\n",
        "**Key Points**\n",
        "\n",
        "-  Combines pairwise and pointwise approaches for robust response selection.\n",
        "-  Offers flexibility with two possible workflows to suit different needs.\n",
        "-  Prioritizes response quality and provides explanations to support user confidence."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ac1c136a334"
      },
      "outputs": [],
      "source": [
        "def rank_responses(instruction: str, context: str, responses: list[str]) -> tuple:\n",
        "    \"\"\"Takes the instruction, context and a variable number of responses as input, and returns the best performing response as well as its associated\n",
        "    human-readable pointwise quality metrics for the configured criteria in the above functions.\n",
        "    The process consists of two steps:\n",
        "    1. Selecting the best response by using Pairwise comparisons between the responses for the user specified metric ( e.g. Q & A)\n",
        "    2. Doing pointwise evaluation of the best response and returning human readable quality metrics and explanation along with the best response.\n",
        "    \"\"\"\n",
        "    cmp_f = partial(\n",
        "        pairwise_greater, instruction, context, PROJECT_ID, LOCATION, experiment_name\n",
        "    )\n",
        "    cmp_greater = partial(greater, cmp_f)\n",
        "\n",
        "    pairwise_best_response = max(responses, key=functools.cmp_to_key(cmp_greater))\n",
        "    pointwise_metric = pointwise_eval(instruction, context, [pairwise_best_response])\n",
        "    qa_metrics = pointwise_metric.metrics_table[\n",
        "        [\n",
        "            col\n",
        "            for col in pointwise_metric.metrics_table.columns\n",
        "            if (\"question_answering\" in col) or (\"groundedness\" in col)\n",
        "        ]\n",
        "    ].to_dict(\"records\")[0]\n",
        "\n",
        "    return pairwise_best_response, qa_metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BY1nfXrqRxVX"
      },
      "source": [
        "### Load the Gemini 2.0 model\n",
        "\n",
        "Here we load the model, and assign a temperature value in the range `0.3` to `1.0` and configure it to generate multiple responses. A higher temperature value is critical, even for use cases where creativity is less important like Q & A: since de-correlated responses would mean if the model gets it wrong with the top choice for one response, it has a possibility of getting it right with one of the other responses"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e761e89b9065"
      },
      "outputs": [],
      "source": [
        "generation_model = GenerativeModel(\"gemini-2.0-flash\")\n",
        "generation_config = GenerationConfig(\n",
        "    temperature=0.4, max_output_tokens=512, candidate_count=num_responses\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8669160b92b9"
      },
      "source": [
        "### Prompt Gemini\n",
        "Now we prompt Gemini to generate multiple slightly de-correlated responses based on the above configuration. Multiple responses will be generated in single call."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3fb5e3ae9faf"
      },
      "outputs": [],
      "source": [
        "instruction_qa = \"Please answer the following question based on the context provided. Question: what is the correct process of fixing your tires?\"\n",
        "context_qa = (\n",
        "    \"Context:\\n\"\n",
        "    \"the world is a magical place and fixing tires is one of those magical tasks. According to the Administration and Association (TIA), the only method to properly repair a tire puncture is to fill the injury with a repair stem and back the stem with a repair patch. This is commonly known as a combination repair or a patch/plug repair.\"\n",
        ")\n",
        "prompt_qa = instruction_qa + \"\\n\" + context_qa + \"\\n\\nAnswer:\\n\"\n",
        "responses = [\n",
        "    candidate.text\n",
        "    for candidate in generation_model.generate_content(\n",
        "        contents=prompt_qa,\n",
        "        generation_config=generation_config,\n",
        "    ).candidates\n",
        "]\n",
        "\n",
        "prompt_qa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c54c44f3f0b5"
      },
      "source": [
        "Here we use the `rank_responses()` function to fetch the best selected response as well as its associated quality metrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "20230a87d5c1"
      },
      "outputs": [],
      "source": [
        "best_response, metrics = rank_responses(instruction_qa, context_qa, responses)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6b5b63a55144"
      },
      "source": [
        "Now we print the various generated responses:\n",
        "1. The raw responses generated by Gemini\n",
        "2. The best performing response\n",
        "3. Its associated pointwise quality metrics and explanation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7ba246f23c1f"
      },
      "outputs": [],
      "source": [
        "for ix, response in enumerate(responses, start=1):\n",
        "    print(f\"Response no. {ix}: \\n {response}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8321faa562f0"
      },
      "outputs": [],
      "source": [
        "print(best_response)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "88458eafb3f9"
      },
      "outputs": [],
      "source": [
        "metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2871493acdd3"
      },
      "source": [
        "## Cleaning Up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial.\n",
        "\n",
        "Otherwise, you can delete the individual resources you created in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "86bbf7dc97ff"
      },
      "outputs": [],
      "source": [
        "aiplatform.init(project=PROJECT_ID, location=LOCATION)\n",
        "experiment = aiplatform.Experiment(experiment_name)\n",
        "experiment.delete()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "enhancing_quality_and_explainability_with_eval.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
