Prompt Based Segmentation

This deep learning model is based on CLIPSeg, is used to create a binary mask based on the provided prompt. This model works as a zero-shot segmentation model and can segment any target object without any fine-tuning. It uses CLIP's pre-trained knowledge of image and prompt relationships to segment images based on a user provided text or visual prompt. Visual prompting could be helpful, where it is hard to describe an image through text prompt due to the complex features in the image, by directly providing the image as prompt for generating the segmentation mask. 

Using the model
Follow the guide to use the model. Before using this model, ensure that the supported deep learning libraries are installed. For more details, check Deep Learning Libraries Installer for ArcGIS.

Fine-tuning the model
This model cannot be fine-tuned using ArcGIS tools.

Input
8-bit RGB imagery.

Output
A classified raster with the prompt as a class.

Applicable geographies
This model is expected to work well globally.

Model architecture
The implementation is based on CLIPSeg.

Accuracy metrics
Refer to the Lüddecke, T., & Ecker, A. S. (2021, December 18). Image segmentation using text and image prompts .

Sample results
Here are a few results from the model.