{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uxCkB_DXTHzf"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hny4I-ODTIS6"
      },
      "source": [
        "# Image Segmentation on Vertex AI\n",
        "\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/image_segmentation.ipynb\">\n",
        "      <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fvision%2Fgetting-started%2Fimage_segmentation.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\"><br> Run in Colab Enterprise\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/getting-started/image_segmentation.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/image_segmentation.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/image_segmentation.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/image_segmentation.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/image_segmentation.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/5a/X_icon_2.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/image_segmentation.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/getting-started/image_segmentation.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XHQ6zSOLt102"
      },
      "source": [
        "| Author |\n",
        "| --- |\n",
        "| [Jorj Ismailyan](https://github.com/jismailyan-google) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-nLS57E2TO5y"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Vertex Image Segmentation brings Google's state of the art segmentation models to developers as a scalable and reliable service.\n",
        "\n",
        "With Image Segmentation, developers can choose from five different modes to segment images and build AI products, including with a **text prompt** and **interactive** mode.\n",
        "\n",
        "Learn more about [Image Segmentation on Vertex AI](https://docs.google.com/document/d/1y5H_m29zGM3Xt6ba2lMw_di6bpbvtQagpU-xY30Kx78/preview?hgd=1&resourcekey=0-_-4WVkfl0oS3nfBwIEhWWQ&tab=t.0).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXsvgIuwTPZw"
      },
      "source": [
        "### Objectives\n",
        "\n",
        "In this notebook, you will be exploring the features of Vertex Image Segmentation using the Vertex AI Python SDK. You will\n",
        "\n",
        "- Segment the foreground or background of an object\n",
        "  - Create a product image by removing the background\n",
        "  - Change the background color of an image\n",
        "- Control the generated mask by configuring dilation\n",
        "- Use an open-vocabulary text prompt to perform:\n",
        "  - Object detection\n",
        "  - Instance segmentation\n",
        "- Draw a scribble to guide segmentation\n",
        "  - Perform point-to-mask segmentation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ewGtTNOo_63z"
      },
      "source": [
        "### Access\n",
        "\n",
        "Request access for your Google Cloud project by visiting the [Image Segmentation model card](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/image-segmentation-001) in Vertex AI Model Garden."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "skXAu__iqks_"
      },
      "source": [
        "### Costs\n",
        "\n",
        "- This notebook uses billable components of Google Cloud:\n",
        "  - Vertex AI\n",
        "\n",
        "- Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mvKl-BtQTRiQ"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B-UOCMvJdmlq"
      },
      "source": [
        "### Install Vertex AI SDK for Python (Jupyter only)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u5lOntr-doIT"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opUxT_k5TdgP"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, run the following cell to authenticate your environment. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vbNgv4q1T2Mi"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# Additional authentication is required for Google Colab\n",
        "if \"google.colab\" in sys.modules:\n",
        "    # Authenticate user to Google Cloud\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybBXSukZkgjg"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and enable the Vertex AI API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "q7YvbXXdtzDT"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your project id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param [\"asia-northeast1\", \"asia-northeast3\", \"asia-southeast1\", \"europe-west1\", \"europe-west2\", \"europe-west3\", \"europe-west4\", \"europe-west9\", \"northamerica-northeast1\", \"us-central1\", \"us-east4\", \"us-west1\", \"us-west4\"]\n",
        "\n",
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)\n",
        "\n",
        "print(f\"Vertex AI SDK client initialized on project {PROJECT_ID} in {LOCATION}.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "Ju_PctW22NUl"
      },
      "outputs": [],
      "source": [
        "# @title Import libraries and define helper functions\n",
        "# @markdown Run this cell before proceeding to import libraries and define\n",
        "# @markdown utility functions.\n",
        "import imghdr\n",
        "import io\n",
        "import random\n",
        "\n",
        "from PIL import Image, ImageDraw\n",
        "from google.colab import files\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from vertexai.preview.vision_models import (\n",
        "    ImageSegmentationModel,\n",
        "    ImageSegmentationResponse,\n",
        ")\n",
        "from vertexai.preview.vision_models import Image as Vertex_Image\n",
        "\n",
        "\n",
        "# Displays PIL images in a row.\n",
        "def display_row(images: list, figsize: tuple[int, int] = (12, 12)):\n",
        "    fig, axes = plt.subplots(1, len(images), figsize=figsize, squeeze=False)\n",
        "\n",
        "    for i, ax in enumerate(axes.ravel()):\n",
        "        ax.imshow(images[i].convert(\"RGBA\"))\n",
        "        ax.axis(\"off\")\n",
        "\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "# Extracts masks from the response and overlays them onto the base image.\n",
        "def overlay_masks(\n",
        "    input_image: Image.Image, response: ImageSegmentationResponse\n",
        ") -> Image.Image:\n",
        "    # Make the original image grayscale to keep overlayed masks visible.\n",
        "    overlayed_image = input_image.copy().convert(\"L\").convert(\"RGB\")\n",
        "\n",
        "    for mask in response.masks:\n",
        "        mask_pil = mask._pil_image\n",
        "        # Gives the mask a distinct color and transparent background.\n",
        "        color = (\n",
        "            random.randint(0, 255),\n",
        "            random.randint(0, 255),\n",
        "            random.randint(0, 255),\n",
        "            128,\n",
        "        )\n",
        "        colored_mask = Image.new(\"RGBA\", mask_pil.size, color)\n",
        "        colored_mask = Image.composite(\n",
        "            colored_mask, Image.new(\"RGBA\", mask_pil.size), mask_pil\n",
        "        )\n",
        "\n",
        "        overlayed_image.paste(colored_mask, (0, 0), colored_mask)\n",
        "\n",
        "    return overlayed_image\n",
        "\n",
        "\n",
        "# Calculates the bounding box coordinates of the masked area in a mask image.\n",
        "\n",
        "\n",
        "def get_bounding_box(mask: Image.Image) -> tuple | None:\n",
        "    mask_array = np.array(mask.convert(\"L\")) > 0\n",
        "    rows, cols = np.nonzero(mask_array)\n",
        "    if rows.size == 0:\n",
        "        return None\n",
        "\n",
        "    x1 = np.min(cols)\n",
        "    y1 = np.min(rows)\n",
        "    x2 = np.max(cols)\n",
        "    y2 = np.max(rows)\n",
        "\n",
        "    return (x1, y1, x2 + 1, y2 + 1)\n",
        "\n",
        "\n",
        "# Converts a segmentation response to labeled bounding boxes.\n",
        "def get_labeled_boxes(response: ImageSegmentationResponse) -> list:\n",
        "    labeled_boxes = []\n",
        "    for mask in response.masks:\n",
        "        bounding_box = get_bounding_box(mask._pil_image)\n",
        "        if bounding_box:\n",
        "            entity_label = mask.labels[0]\n",
        "            score = round(float(entity_label.score), 3)\n",
        "            labeled_box = (entity_label.label, score, bounding_box)\n",
        "            labeled_boxes.append(labeled_box)\n",
        "\n",
        "    return labeled_boxes\n",
        "\n",
        "\n",
        "# Draws bounding boxes on a base image around each generated mask.\n",
        "def draw_bounding_boxes(\n",
        "    base_image: Image.Image, response: ImageSegmentationResponse\n",
        ") -> Image.Image:\n",
        "    bbox_image = base_image.copy()\n",
        "    labeled_boxes = get_labeled_boxes(response)\n",
        "    color = \"green\"\n",
        "    draw = ImageDraw.Draw(bbox_image)\n",
        "    for box in labeled_boxes:\n",
        "        bounding_box = box[2]\n",
        "        draw.rectangle(bounding_box, outline=color, width=2)\n",
        "\n",
        "        text_label = f\"{box[0]}: {box[1]}\"\n",
        "        text_width = (len(text_label) * 5) + 3  # Add 2 for padding\n",
        "        text_height = 12\n",
        "        label_x = bounding_box[0]\n",
        "        label_y = bounding_box[1]  # Position label above the box\n",
        "\n",
        "        # Draw a filled rectangle as the background for the label\n",
        "        draw.rectangle(\n",
        "            (label_x, label_y, label_x + text_width, label_y + text_height),\n",
        "            fill=color,\n",
        "        )\n",
        "        draw.text((label_x + 2, label_y), text_label, fill=\"white\")\n",
        "\n",
        "    return bbox_image\n",
        "\n",
        "\n",
        "IMAGE_SEGMENTATION_MODEL = \"image-segmentation-001\"\n",
        "segmentation_model = ImageSegmentationModel.from_pretrained(IMAGE_SEGMENTATION_MODEL)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R45VRKWInfQQ"
      },
      "source": [
        "## Select an image to segment\n",
        "\n",
        "Run this cell to enable and select the `Choose files` button.\n",
        "You can then select an image file from your local device to upload."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I9caIrZ7Dek1"
      },
      "outputs": [],
      "source": [
        "images = files.upload()\n",
        "raw_image_bytes = list(images.values())[0]\n",
        "\n",
        "BASE_IMAGE = Vertex_Image(raw_image_bytes)\n",
        "BASE_IMAGE_PIL = BASE_IMAGE._pil_image\n",
        "w, h = BASE_IMAGE_PIL.size\n",
        "\n",
        "display_row([BASE_IMAGE_PIL], (18, 18))\n",
        "print(f\"Base image width x height: {w} x {h}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fU32286ooc8Q"
      },
      "source": [
        "## Segment images using different modes\n",
        "\n",
        "You can generate image masks with different Image Segmentation features by setting the `mode` field to one of the available options:\n",
        "* **Foreground**: Generate a mask of the segmented foreground of the image.\n",
        "* **Background**: Generate a mask of the segmented background of the image.\n",
        "* **Semantic**: Select the items in an image to segment from a set of 194 classes.\n",
        "* **Prompt**: Use an open-vocabulary text prompt to guide the image segmentation.\n",
        "* **Interactive**: Draw a rough mask to guide the model segmentation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mBLJtICO8iMQ"
      },
      "source": [
        "### Foreground segmentation request\n",
        "\n",
        "This section will explores how to edit images using different `edit_mode` and `mask_mode` parameter options."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c9N8l0oo_cWs"
      },
      "outputs": [],
      "source": [
        "gcs_uri = None  # gs:// path to the input image\n",
        "mode = \"foreground\"  # Segmentation mode [foreground,background,semantic,prompt,interactive]\n",
        "prompt = None  # Prompt to guide segmentation for `semantic` and `prompt` modes\n",
        "scribble = None  # Input scribble for `interactive` segment mode\n",
        "mask_dilation = (\n",
        "    None  # Optional mask dilation for thin objects. Numeric value between 0 and 1.\n",
        ")\n",
        "max_predictions = (\n",
        "    None  # Optional maximum predictions limit for prompt mode. Unlimited by default.\n",
        ")\n",
        "confidence_threshold = (\n",
        "    None  # Optional confidence limit for prompt/background/foreground modes.\n",
        ")\n",
        "binary_color_threshold = (\n",
        "    -1  # Optional binary thresholding level, or -1 to get grayscale \"soft\" masks.\n",
        ")\n",
        "\n",
        "response = segmentation_model.segment_image(\n",
        "    BASE_IMAGE,\n",
        "    prompt,\n",
        "    scribble,\n",
        "    mode,\n",
        "    mask_dilation,\n",
        "    max_predictions,\n",
        "    confidence_threshold,\n",
        "    binary_color_threshold,\n",
        ")\n",
        "MASK_PIL = response.masks[0]._pil_image\n",
        "display_row([BASE_IMAGE_PIL, MASK_PIL], (18, 18))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a5yLGWLwOVIJ"
      },
      "source": [
        "#### Background removal\n",
        "Use the foreground segmentation mask you created above to make the image background transparent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-5K0CSb0twPO"
      },
      "outputs": [],
      "source": [
        "# Creates an empty transparent background.\n",
        "transparent_background = Image.new(\"RGBA\", BASE_IMAGE_PIL.size, (128, 128, 128, 255))\n",
        "\n",
        "# Uses the mask to cut and paste the foreground object in the original image\n",
        "# onto the transparent background.\n",
        "transparent_background.paste(BASE_IMAGE_PIL, mask=MASK_PIL)\n",
        "transparent_background.putalpha(MASK_PIL)\n",
        "\n",
        "display_row([transparent_background], (18, 18))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEUZeTtyO01R"
      },
      "source": [
        "#### Change background color"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k-WiqiFJO448"
      },
      "outputs": [],
      "source": [
        "# RGBA color light blue\n",
        "color = (141, 224, 254, 255)\n",
        "gray_background = Image.new(\"RGBA\", BASE_IMAGE_PIL.size, color)\n",
        "gray_background.paste(BASE_IMAGE_PIL, mask=MASK_PIL)\n",
        "\n",
        "display_row([gray_background], (18, 18))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "--7rofOb95hT"
      },
      "source": [
        "### Background segment mode\n",
        "\n",
        "Generate background masks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JVtC3lFAGoAu"
      },
      "outputs": [],
      "source": [
        "response = segmentation_model.segment_image(\n",
        "    BASE_IMAGE, mode=\"background\", mask_dilation=None, binary_color_threshold=-1\n",
        ")\n",
        "MASK_PIL = response.masks[0]._pil_image\n",
        "display_row([BASE_IMAGE_PIL, MASK_PIL], (18, 18))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U9pfcnNsGtcv"
      },
      "source": [
        "### Semantic segment mode\n",
        "\n",
        "Specify the objects to segment from the set of 194 classes. The full set is available in the Appendix section at the end of this tutorial. You can specify multiple classes by delimiting with commas, e.g. `prompt=\"cat, dog\"`\n",
        "\n",
        "The semantic segmenter will return a single prediction containing the generated mask. If the classes in the prompt are detected, they are masked in white pixels and the background will be black. If the requested classes are not detected in the image, the whole mask will be black."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Aar3Pn3yG75T"
      },
      "outputs": [],
      "source": [
        "response = segmentation_model.segment_image(\n",
        "    BASE_IMAGE,\n",
        "    prompt=\"motorcycle, bus\",\n",
        "    mode=\"semantic\",\n",
        "    mask_dilation=None,\n",
        "    binary_color_threshold=-1,\n",
        ")\n",
        "MASK_PIL = response.masks[0]._pil_image\n",
        "display_row([BASE_IMAGE_PIL, MASK_PIL], (18, 18))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CO4q2sacIydg"
      },
      "source": [
        "### Prompt instance segmentation mode\n",
        "\n",
        "You can use Prompt mode to perform detection and segmentation on many instances of your suggested objects. The response can generate multiple masks, along with one or more associated labels for each mask. Each label also contains an confidence score. Only objects matching labels specified in the request prompt are detected and segmented. The prompt is completely open-vocabulary, it is not limited to any class set.\n",
        "\n",
        "**Recommended**:\n",
        "* Use the confidence_threshold and max_predictions parameters to filter and limit results\n",
        "* You can request multiple items be detected by separating them with commas. Hundreds of classes can be set in a single prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "unrHSqhqHmlI"
      },
      "outputs": [],
      "source": [
        "prompt = \"green watermelon, cantaloupe, price tag\"\n",
        "threshold = 0.1\n",
        "\n",
        "response = segmentation_model.segment_image(\n",
        "    BASE_IMAGE,\n",
        "    prompt=prompt,\n",
        "    mode=\"prompt\",\n",
        "    mask_dilation=None,\n",
        "    max_predictions=None,\n",
        "    confidence_threshold=threshold,\n",
        "    binary_color_threshold=-1,\n",
        ")\n",
        "\n",
        "count = str(len(response.masks))\n",
        "print(f\"Detected {count} objects at threshold {threshold}.\")\n",
        "\n",
        "bbox_image = draw_bounding_boxes(BASE_IMAGE_PIL, response)\n",
        "overlayed_image = overlay_masks(BASE_IMAGE_PIL, response)\n",
        "display_row([BASE_IMAGE_PIL, bbox_image, overlayed_image], figsize=(25, 25))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UuaLTarf-hvO"
      },
      "source": [
        "## Appendix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ZzRAwQ0dIjT"
      },
      "source": [
        "### Semantic segmentation classes\n",
        "\n",
        "| Class ID | Class ID | Class ID | Class ID |\n",
        "| --- | --- | --- | --- |\n",
        "|   `backpack`  |   `broccoli`  |   `road`  |   `mountain_hill`   |\n",
        "|   `umbrella`  |   `carrot`  |   `snow`  |   `rock`    |\n",
        "|   `bag` |   `hot_dog` |   `sidewalk_pavement` |   `frisbee`   |\n",
        "|   `tie` |   `pizza` |   `runway`  |   `skis`    |\n",
        "|   `suitcase`  |   `donut` |   `terrain` |   `snowboard`   |\n",
        "|   `case`  |   `cake`  |   `book`  |   `sports_ball`   |\n",
        "|   `bird`  |   `fruit_other` |   `box` |   `kite`    |\n",
        "|   `cat` |   `food_other`  |   `clock` |   `baseball_bat`    |\n",
        "|   `dog` |   `chair_other` |   `vase`  |   `baseball_glove`    |\n",
        "|   `horse` |   `armchair`  |   `scissors`  |   `skateboard`    |\n",
        "|   `sheep` |   `swivel_chair`  |   `plaything_other` |   `surfboard`   |\n",
        "|   `cow` |   `stool` |   `teddy_bear`  |   `tennis_racket`   |\n",
        "|   `elephant`  |   `seat`  |   `hair_dryer`  |   `net`   |\n",
        "|   `bear`  |   `couch` |   `toothbrush`  |   `base`    |\n",
        "|   `zebra` |   `trash_can` |   `painting`  |   `sculpture`   |\n",
        "|   `giraffe` |   `potted_plant`  |   `poster`  |   `column`    |\n",
        "|   `animal_other`  |   `nightstand`  |   `bulletin_board`  |   `fountain`    |\n",
        "|   `microwave` |   `bed` |   `bottle`  |   `awning`    |\n",
        "|   `radiator`  |   `table` |   `cup` |   `apparel`   |\n",
        "|   `oven`  |   `pool_table`  |   `wine_glass`  |   `banner`    |\n",
        "|   `toaster` |   `barrel`  |   `knife` |   `flag`    |\n",
        "|   `storage_tank`  |   `desk`  |   `fork`  |   `blanket`   |\n",
        "|   `conveyor_belt` |   `ottoman` |   `spoon` |   `curtain_other`   |\n",
        "|   `sink`  |   `wardrobe`  |   `bowl`  |   `shower_curtain`    |\n",
        "|   `refrigerator`  |   `crib`  |   `tray`  |   `pillow`    |\n",
        "|   `washer_dryer`  |   `basket`  |   `range_hood`  |   `towel`   |\n",
        "|   `fan` |   `chest_of_drawers`  |   `plate` |   `rug_floormat`    |\n",
        "|   `dishwasher`  |   `bookshelf` |   `person`  |   `vegetation`    |\n",
        "|   `toilet`  |   `counter_other` |   `rider_other` |   `bicycle`   |\n",
        "|   `bathtub` |   `bathroom_counter`  |   `bicyclist` |   `car`   |\n",
        "|   `shower`  |   `kitchen_island`  |   `motorcyclist`  |   `autorickshaw`    |\n",
        "|   `tunnel`  |   `door`  |   `paper` |   `motorcycle`    |\n",
        "|   `bridge`  |   `light_other` |   `streetlight` |   `airplane`    |\n",
        "|   `pier_wharf`  |   `lamp`  |   `road_barrier`  |   `bus`   |\n",
        "|   `tent`  |   `sconce`  |   `mailbox` |   `train`   |\n",
        "|   `building`  |   `chandelier`  |   `cctv_camera` |   `truck`   |\n",
        "|   `ceiling` |   `mirror`  |   `junction_box`  |   `trailer`   |\n",
        "|   `laptop`  |   `whiteboard`  |   `traffic_sign`  |   `boat_ship`   |\n",
        "|   `keyboard`  |   `shelf` |   `traffic_light` |   `slow_wheeled_object`   |\n",
        "|   `mouse` |   `stairs`  |   `fire_hydrant`  |   `river_lake`    |\n",
        "|   `remote`  |   `escalator` |   `parking_meter` |   `sea`   |\n",
        "|   `cell phone`  |   `cabinet` |   `bench` |   `water_other`   |\n",
        "|   `television`  |   `fireplace` |   `bike_rack` |   `swimming_pool`   |\n",
        "|   `floor` |   `stove` |   `billboard` |   `waterfall`   |\n",
        "|   `stage` |   `arcade_machine`  |   `sky` |   `wall`    |\n",
        "|   `banana`  |   `gravel`  |   `pole`  |   `window`    |\n",
        "|   `apple` |   `platform`  |   `fence` |   `window_blind`    |\n",
        "|   `sandwich`  |   `playingfield`  |   `railing_banister`  |       |\n",
        "|   `orange`  |   `railroad`  |   `guard_rail`  |       |"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "image_segmentation.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
