Label Studio x Segment Anything in Hugging Face Spaces
![](png/5860d4b8cb3a64a9765995891c7c0280f3f22584-1440x756_z1gshhp.png)
The Label Studio community is excited to announce the release of the Segment Anything Machine Learning Backend on Hugging Face Spaces! This new release delivers the groundbreaking power of Segment Anythingâs generalized object detection knowledge, using the MobileSAM build of the model as a Hugging Face Spaces application that will make it even easier to launch and connect to your Label Studio instance.
What is the Segment Anything Model?
Segment Anything is a model the Meta AI research team developed that âcan âcut outâ any object, in any image, with a single click.â One of the most compelling features of SAM is that it âis a promptable segmentation system with zero-shot generalization to unfamiliar objects and images, without the need for additional training.â
![](png/6e5157d89e25960e7de40c14720f7ddf9a3b40af-1319x794_zbyoxt.png)
Segment Anything Model Selecting a Dog with a Bounding Box Constraint
What this means at a higher level is that SAM has a more generalized notion of what objects are, allowing it to identify and segment objects in an image even if it has never previously encountered them in training. This sort of zero-shot generalization, combined with powerful keypoint and bounding-box cueing, makes SAM an ideal tool for pre-annotating new image segmentation datasets.
You can learn more about the model at the official SAM website.
What is Hugging Face Spaces?
From the Hugging Face Spaces documentation: "Spaces offers a simple way to host ML demo apps directly on your or your organizationâs. This flexible platform allows you to create your ML portfolio, showcase your projects at conferences or to stakeholders, and work collaboratively with others in the ML ecosystem." Beyond that, Hugging Face Spaces gives users a way to build and share ML applications collaboratively with the broader ML community.
For example, the Label Studio community launched its own Label Studio Space earlier this year. This space allowed users to try out Label Studio in a sandbox environment or replicate the Label Studio Space to run in their environment. With the addition of permanent storage in Spaces, you can even host a production-ready Label Studio Instance entirely in Hugging Face Spaces.
Now, weâre growing the portfolio of Label Studio Spaces with the release of the âLabel Studio x Segment Anythingâ space. This article will guide you through launching Label Studio and Segment Anything Model instances to build an object labeling workflow.
Launch & Configure Label Studio
The fastest way to get started is to duplicate the Label Studio Space into your Hugging Face account. Navigate to the official Label Studio application space, expand the menu in the Spaces toolbar, and select Duplicate this Space. You'll be prompted to fill out some basic information, and then after you hit the Duplicate Space button, youâll be ready to go. Once the interface is running, you can create a new login and get started with a new Label Studio demonstration environment.
![](png/6e1cc4305fb66a9fb6c4c3c91a3a55beead79b0e-1288x886_1x7jhx.png)
Duplicating the Official Label Studio Space
To run a secure instance of Label Studio with persistent storage, check out the article from earlier this year, Kickstart your Label Studio Annotation Project on Hugging Face Spaces, for complete details.
To give the Segment Anything Model access to your image data, you must enable file sharing from Label Studio. Select Settings in your Label Studio Space, then scroll down to Variables and Secrets, and add the following variables:
LABEL_STUDIO_LOCAL_FILES_SERVING_ENABLED=true
LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT=/label-studio/data/images
![](png/e975546963042efb03e9b9b4ae57001144aea1b0-1288x751_ztpnne.png)
Enabling Local File Serving in Label Studio
The first enables local file serving, allowing SAM to download files from Label Studio. The second specifies the location where Label Studio should serve files. Use the setting above if youâre running Label Studio in demo mode. If you use permanent storage, using the instructions from the Kickstart article, set that variable to /data/images
.
Get your Label Studio Endpoint and API Key
You will need to configure your SAM instance with information about how to connect to Label Studio. To get the direct URL to your Label Studio instance, select the Spaces application menu, choose Embed this Space, and make a note of the Direct URL value.
![](png/5e4ce9ac652012e58162bf6543e22d4baacfb184-1288x886_z1k554h.png)
Copying the Label Studio URL
To get the Label Studio API key, log into Label Studio, click the user settings in the right-hand corner of the application, select Accounts & Settings from the dropdown menu, and note the Access Token.
![](png/692c20f2972ced74c7a098c1eea303b0b4f03784-1288x886_z2krh6v.png)
Copying the Label Studio Access Token
With these two values, youâre ready to launch your SAM Space!
Launch and Configure the SAM ML Backend
Now itâs time to launch the Segment Anything Space. Navigate to the official Label Studio Space for Segment Anything, select the Space application menu, and choose Duplicate this Space. Just as before, youâll be prompted to fill out some basic information, and then after you hit the Duplicate Space button, your copy of the model will start to build.
As the model is building, navigate to the âSettingsâ of the space and set the following secrets with the values collected in Step 2:
LABEL_STUDIO_HOST=<YOUR_DIRECT_URL_HERE>
LABEL_STUDIO_ACCESS_TOKEN=<YOUR_ACCESS_TOKEN_HERE>
![](png/02bf92fa6d20b267e9619b36269a4dc45d65188f-1288x886_11hkb7.png)
Setting the LABEL_STUDIO_HOST Variable
Wait for the space to build and launch. Youâll know itâs ready when the application reports:
{"model_class":"SamMLBackend","status":"UP"}
![](png/ddfd0c62b5d2eee80f70b34d02225d6ad9cb7442-656x209_1bzow3.png)
Verifying the Segment Anything Model is Running
Get the SAM URL by selecting the Spaces application menu and clicking Embed this Space. Make a note of the Direct URL value.
Set Up Your Labeling Project
Upon your first login, youâll be greeted by Heidi the Opossum, asking you to create a new project. Select Create Project.
![](png/ee586670d45adb61ccc119c800f1a8768f314e2e-1046x650_bfqmj.png)
"Heidi doesn't see any projects here."
In the Create Project dialog, you will see three tabs. In the first tab, enter the name of your project (for example, Segment Anything), then select Labeling Setup. Youâll be presented with pre-configured labeling templates, but we will import a custom template for this project. Select the Custom Template option.
![](png/fe3daa13b23ff9c81127515f785870a50410cbb4-1046x1076_z10kwr4.png)
The Labeling Setup Interface
Youâll be presented with a template editing interface. Select Code, delete any existing code, and then paste the following configuration into the text field.
<View>
<Image name="image" value="$image" zoom="true"/>
<Header value="Brush Labels"/>
<BrushLabels name="tag" toName="image">
<Label value="Dog" background="#FF0000"/>
<Label value="Possum" background="#0d14d3"/>
</BrushLabels>
<Header value="Keypoint Labels"/>
<KeyPointLabels name="tag2" toName="image" smart="true">
<Label value="Dog" smart="true" background="#000000" showInline="true"/>
<Label value="Possum" smart="true" background="#000000" showInline="true"/>
</KeyPointLabels>
<Header value="Rectangle Labels"/>
<RectangleLabels name="tag3" toName="image" smart="true">
<Label value="Dog" background="#000000" showInline="true"/>
<Label value="Possum" background="#000000" showInline="true"/>
</RectangleLabels>
</View>
Note that for this example, we are interested in labeling images with "Dogs" and "Possums." Feel free to replace these tag values with whatever application youâre interested in and with as many tags as you wish.
The labeling configuration includes brush, keypoint, and rectangle label interface elements. The keypoint and rectangle labels will send inference requests to the SAM ML Backend. The SAM backend doesnât support smart brush labeling, but you can use the brush tool to refine automatically generated keypoint labels.
Youâll be presented with a sample labeling interface as you enter the code. When you are satisfied with the interface, select Save.
![](png/f2dd9bf411bf834a70d15d38ca2264053e51e8bc-1046x1076_18ctnv.png)
The Final Custom Labeling Interface
Connect Label Studio and SAM
Select the menu for Projects, then select Segment Anything (or whatever name you set for your project). Select Settings, then Machine Learning.
![](png/3ff1089d23a425af0dcfa52478b5e54befd68607-1046x599_z1we5n0.png)
The Machine Learning Configuration Panel
Select Add Model, and fill in the Title and URL details. The URL will be the address for the Segment Anything space you set up. Toggle the switch for Use for interactive preannotations to on, then select Validate and Save.
![](png/75812647a5603b96a46160b7bca12106434f7593-1288x913_2fjezr.png)
Adding and Validating the Segment Anything Model
Select Save, then select the project name in the navigator (right after Projects, in this example, Projects/Segment Anything) to return to the task interface.
Label Studio will present you with the task management interface, where youâll see a screen indicating you havenât imported any data yet.
![](png/5f0f2cd068f9979bd8f5814aa221419a19680eab-1046x1076_pccja.png)
"Looks like you have not imported any data yet."
Select Go to import and import your first project images. Select Upload Files and choose any images you wish to use. For this example, we will use pictures of dogs and possums, but you can use any dataset you want.
Start Labeling
With your images imported, itâs time to start labeling! Select Label all Tasks to get started!
Label with Keypoints
After selecting Label all Tasks, youâll be presented with the task labeling interface. Youâll have your image labeling task on the upper left side of the interface. Below it will be three rows of labels. The top row is for brush labeling, the second is for keypoint labeling, and the third is for bounding box labeling. To the right of the image, youâll see a toolbar. At the bottom, you will see some navigation buttons. From the navigation buttons, ensure Auto-annotation is toggled to the on position, then make sure Auto accept annotation suggestions are unselected at the top of the image.
![](png/6bcbad69b393a906ccfade2dbdc2e411e3cd710b-1046x1076_z193un9.png)
Overview of the Labeling Interface
We will begin with automated keypoint labeling. In the toolbar to the right, select the purple smart selection tool and click on it until the three purple dots are selected (this selects the smart keypoint tool).
![](png/7c48e445278e591cbf7d6cea36a96048b210600a-1046x400_cdgww.png)
Selecting the Smart Keypoint tool
Now select the label you want in the second row (pro-tip, use the hot-key selection associated with the label to pick the label you want, in this case, 3). When the label is highlighted, select anywhere in the image to drop a keypoint and automatically trigger the SAM backend. The model may take a few seconds to return the prediction, depending on your hardware.
![](png/f830ad0bf571cb9e64c3ad8fa9031264ea1d2b98-1046x1076_zr4vd0.png)
Using a Smart Keypoint to Select an Object Mask
If youâre satisfied with the segmentation mask, click the check mark next to the Auto-Annotation tool. If youâre not, repeat the process by selecting the keypoint tool and clicking the region you want to add to the annotation (this is where using hotkeys comes in handy, so in this instance, you would press 3 to select the Dog keypoint annotation then click on the region you want to refine).
![](png/3d1f963ff9a1b29a9aa8473c2ee147bf6f3754d7-1046x1076_z14cb0v.png)
Refining the Selection with Additional Keypoints
If the model has over-selected a region, you can make negative selections by holding down the option key while you drop a keypoint. Option-select will create a negative affinity and remove unwanted parts of the annotation. Select the checkmark when youâre satisfied with the selection, and the labeling interface will merge your keypoint selections into one mask. Note that the model also gives a confidence score for the selection, in this case 97%.
![](png/31b8f3fce833614f1053a4e83c392ca09dac7905-1046x1076_2jinfu.png)
The Final Image Annotation
If youâre happy with the label, select Submit and advance automatically to the next labeling task.
Label with Bounding Boxes
In the following example, this time with a possum, begin by selecting the appropriate smart annotation tool, in this case, the rectangle bounding box.
![](png/39bed44bb6b2d455754ef672b55e8a665b6c8928-1046x1076_dluqe.png)
Selecting the Smart Rectangle Tool
Then select the rectangle tag you want to apply from the third row, in this case, Possum (with the associated hot-key 6). Drag a rectangle around the subject youâre interested in, and the SAM model will make a prediction based on that region.
![](png/d91476f4019577c66321797d7fbd4d7aa92042c9-1046x1076_2svjzm.png)
Object Mask Using the Smart Rectangle Tool
If youâre satisfied with the chosen region, select the check mark and submit the annotation.
![](png/338b0235653c65eec4ad2dd894de217e9a1ba55e-1046x1076_z10kbbh.png)
Submitting the Final Annotation
Suppose you find (as with this example, with a confidence score of 92%) that the region is over-selected; you should use the keypoint tool to give you more control (with affinity and anti-affinity points) over the selection refinement.
Whatâs Next?
Label Studio plays a critical role in the machine learning pipeline, giving an interface for humans to guide the essential step of labeling and annotating data alongside machine learning systems to speed the process. You can learn more about integrating Label Studio into your machine-learning pipeline in the Label Studio docs. Check out the GitHub repository for complete documentation on the SAM ML Backend.
Once a labeling project is finished, you can export the labels using the âExportâ interface from the project management home. Masks and annotations are exported as a JSON catalog for your ML and data science pipeline.
Shivansh Sharma, an active Label Studio Community member, developed the original SAM ML Backend. If you have projects youâd like to share or want to collaborate with others in launching your labeling project, join the Label Studio Slack Community, where over 10,000 Label Studio users have signed up to share insights and meet with their peers.
Happy Labeling!