Introducing Label Studio, a swiss army knife of data labeling
Professional machine learning tools are both the foundation and scaffolding for building successful ML-based products.
I’ve experienced the lack of tools myself while working at one of the enterprises on a personal virtual assistant project, which was used by around 20 million people. Our team was continually looking for ways to improve quality, handle edge cases, and test hypotheses. That usually required working with unstructured data, labeling it, visually examining and exploring models predictions. We’d hack and glue together a set of internal tools, with tons of boilerplate code made to work once. Needless to say that sharing those tools or trying to extend it or embed it into the main application would be nearly impossible.
Moreover, this process was about quantitative analysis of the product. While trying to improve models, machine learning engineers heavily rely on precision/recall statistics computed on a fixed dataset, ignoring how this dataset aligns with actual real-world data. Eventually, it leads to systematic errors in production, which could be identified only by qualitative analysis — basically looking with your eyes at the model's predictions.
A couple of my friends and I got thinking, can we do better? And this is how Label Studio was born. It is intended to save prototyping/experimenting time for individual machine learning practitioners, as well as reducing the ML product release life cycle for technical teams.
Some of the principals we were following while working on it:
- Making it simple. No complicated configs, and ease of integration into Machine Learning pipelines. Label Studio can be used in different places, depending on the use-case:
- Quickly configurable for many data types. Get the tool ready in 10 minutes. There should be an easy way to switch between labeling texts, audios or images, or even annotating all three types at the same time.
- Machine learning integration. It should be able to integrate with all the plethora of machine learning frameworks and models. There are numerous applications of ML with different constraints, and Label Studio has to be flexible enough to handle them and help, not complicate.
If that sounds entertaining, let’s install it and get you up and running!
Let’s Start
Starting Label Studio is extremely easy:
pip install label-studio
label-studio start my_project --initIt automatically opens up the web app in your browser. Just configure what you want to label and how. That’s it.
Many existing labeling frameworks accept only one data type, and it becomes tedious to learn a new app each time. Right out of the box Label Studio works with Texts, Images, Audios, HTML documents (called Object components), and any imaginable combination of annotation tasks like classification, regression, tagging, spanning, pairwise comparison, object detection, segmentation and so on (defined in the Annotation components).
Let’s explore how you can configure the Label Studio server for your use case
Configuring Labeling Interface
Label Studio’s interface is not pre-built. Instead, you create it yourself, in the same sense as you’d create a webpage. But instead of using HTML tags, you get jsx-based components. Fear not! You don’t need to write JavaScript unless you want to. The components know how to connect based on their type and name. Here is an example of Image classification where the Choices (Annotation component) gets connected to Image (Object component) by specifying its name using the toName attribute.
Note the dollar sign variable $url. It specifies that the value for this component is coming from your dataset, it would expect the record with key url provide the URL for the image. Go ahead to the setup page and play around with the examples that we have in there. Or if you’re not running it locally yet, you can check out the playground.
Currently, there are about 20 components covering different types of labeling, for example, Bbox for the Images or NER for the Text. A full list can found on the website.
Data Import and Export
After configuring what the labeling interfaces should look like, you can import your data. The web import supports multiple formats: JSON, CSV, TSV, and archives consisting of those. Each data point that you want to label is called a task. If your task contains images, you have to host them somewhere and expose URLs in data keys like {"url": "https://labelstud.io/opossum.jpg"} in the example above. The task is then loaded into the labeling interface and waits for you to label it.
Exports are done using label-studio-converter. It’s a library that can take internal Label Studio JSON based format and output either some general-purpose (JSON, CSV, TSV) or model-specific formats like CONLL for textual taggers or Pascal VOC or COCO for computer vision models.
Underlying storage is plain files. Making it very easy to integrate, you only need to put the data in the format that Label Studio can parse. You can find more about the formats on the website.
Alright! Now you have the data in, you know how to label and how to export, let’s go over a few use cases how you can integrate Label Studio into your pipelines.
Features & Use Cases
Label Studio is, first and foremost, a data labeling tool, which makes it very usable for all the essential data labeling activities, like labeling the entire dataset or creating ground truth labels for verification. Besides those basics, here a few other use-cases that are pretty unique to its functionality.
Embed into your app
For some applications, one of the best labels you can get may come from the users of your products. You can ask them to label tasks from scratch, or provide a model prediction and request to adjust it. Then use that to update models and improve application efficiency continually.
The unique feature of Label Studio is, it comes as a frontend NPM package that you can include in your applications. Here is an example:
You can check it live here. Or look at the source of the gist. You can save it locally and open it in your browser!
Machine Learning Integration, connecting the model
You can easily connect your favorite machine learning framework with Label Studio by using the SDK.
That gives you the opportunities to use:
- Auto labeling. Use model predictions for labeling data. It helps to annotate faster by task pre-labeling, as well as using pseudo labeling for further training
- Continual Learning. Continuously annotate and train your model from a stream of data, potentially with changing annotation targets
- Active Learning. Implement techniques for smarter selection of tasks to be labeled
- Prediction Service. Instantly create and deploy REST API prediction service
To learn more about getting up and running, check out the example from the README.
Comparing predictions
You can load multiple predictions from different models architectures or versions into the interface, visually verify, find discrepancies, and edit what your models are predicting:
And Even More
We’ve covered just a handful of use cases that we have found exciting ourselves and where we’ve seen Label Studio can provide an advantage over existing solutions, but while we’re working on the next article, here are a few more ideas for you to explore:
Monitoring model prediction errors
Can you always trust your model predictions? How do you understand when would be a good time to retrain and redeploy a model? You can integrate Label Studio into the model monitoring pipeline. And as in the example above, you can show multiple versions of your model predictions and check if the prediction quality has not decreased.
Human in the (loop) prediction pipeline
For the applications where predictive quality is mission-critical, Label Studio can be used to adjust the model prediction when the model is uncertain about it or acts as a low-precision detector. In these scenarios, the model prediction is first sent to the annotator, and the annotator can manually adjust it.
Collect results from multiple people
Label Studio supports multiple results per task (called completions), which can be very valuable if you need to distribute the same task to numerous annotators and then verify or combine their results. It works similarly to the model prediction visualization with a difference that completion can be set as ground truth and edited.
Incremental dataset labeling
After deploying the model into production or while developing it, you may realize that labeling for one more attribute can enhance the model results. That might be a hypothesis you want to test or a predefined strategy — start with a small number of attributes and add more with time. You can use Label Studio for incremental labeling: modify the labeling config and add new classes on the go.
Next Steps
That is just a start. We’re planning to cover more and more data types and implement more components covering different labeling scenarios.
- Visit the main repository on GitHub
- Explore the templates we’ve created for some popular labeling cases https://labelstud.io/templates/
- Check the tags https://labelstud.io/tags/
- Try out running a demo where you can create your annotation playground persistent within a browser session
We would always be happy to learn more about possible human-in-the-loop use cases and looking forward to implementing them within Label Studio. If you have suggestions and/or feedback, please share it with us by opening an issue on GitHub or joining our growing Slack community!
Happy labeling!
