The world’s leading publication for data science, AI, and ML professionals.

Using OpenAI’s CLIP to Search for Design Patents

Using natural language to search and find over sixty thousand industrial designs that have entered the public domain since 2002

Sample of Expired US Design Patents, Source: USPTO, Image by Author
Sample of Expired US Design Patents, Source: USPTO, Image by Author

Everybody knows about using patents as a way to protect new inventions. Unless you are a patent attorney, you probably only know about one of the three major types, utility patents. However, there are two other types, Design and plant patents. Currently, there are over 10 million granted utility patents, 910 thousand design patents, and almost 32 thousand plant patents.

If you guessed that plant patents are for new varieties of plants, you would be correct! But what are design patents? Here’s the definition from the US Patent and Trademark Office:

"The patent laws provide for the granting of design patents to any person who has invented any new and non-obvious ornamental design for an article of manufacture. The design patent protects only the appearance of an article, but not its structural or functional features. The proceedings relating to granting of design patents are the same as those relating to other patents with a few differences …" – USPTO [1]

One of the differences between utility patents and design patents is the term, the amount of time the owner has a monopoly on the patented item. Currently, the default term for a utility patent is 20 years from when the patent was filed. The default term for a design patent is 15 years from when the patent was granted. Once the term runs out, or if the owner abandons the patent earlier, the work enters the public domain. At that point, the design can be used by anyone, without permission from the owner of the expired patent. Of the 910 thousand design patents, more than half are in the public domain and are free to use.

Overview

In this article, I’ll show you how to use OpenAI‘s new CLIP encoders to perform an image and text-based semantic search on over 67 thousand design patents that have been in the public domain since 2002.

Diagram by Author
Diagram by Author

The system is based on CLIP, an open-source model from OpenAI that performs a semantic search on text and images. The model has two functions, encoding text and images into "embeddings", which are strings of numbers that represent the gist of the original data. The CLIP model was pretrained on 40 million pairs of images with text labels [2] such that the embeddings encoded from the images will be similar to the embeddings encoded from the text labels.

I wrote a Python program to download five years of the weekly Patent Gazette from the USPTO and parse the files to extract design patent names and corresponding images. I then ran all of the captions and images through the CLIP text and image encoders, respectively, and saved the results as two sets of embeddings.

To perform a search, you start by typing in a short query. The system runs the query through the CLIP text encoder and compares the resultant embedding to find the top matches using both the text and image embeddings of the design Patents. The system shows the corresponding captions and images of the top results, which are obtained from an indexed array.

For example, if you search for "folding futon couch", here are the top five results:

Results of CLIP-based Semantic Search, Source: USPTO
Results of CLIP-based Semantic Search, Source: USPTO

In the sections below, I will discuss design patents and describe the Semantic Search process in more detail. A Google Colab is available here so you can try it yourself.

George Bruce, Image Source: wikimedia.org, and his Double Small-Pica Script, Image Source: George Bruce's Son & Co. Both images are in the Public Domain.
George Bruce, Image Source: wikimedia.org, and his Double Small-Pica Script, Image Source: George Bruce’s Son & Co. Both images are in the Public Domain.

Design Patents

As I mentioned above, design patents protect the appearance, not the functional aspects of inventions. The first design patent was granted on November 9, 1842, to George Bruce of New York for a new typeface called Double Small-pica Script [3].

The patent office groups each design patent into one of 33 categories [4], listed below. (Brief aside: at first glance, it seems that there are 35 categories, D1–D34 plus D99 Misc. But oddly, category numbers D31 and D33 are missing from the list. Maybe the USPTO has secret categories for ornamental designs of time machines and perpetual motion machines? 😃 )

Design Patent Categories, Source: USPTO
Design Patent Categories, Source: USPTO

The number of design patents granted annually by the USPTO has grown exponentially since the 1960s. In 2019 there were almost 35,000 design patents granted.

Image by Author, Data Source: USPTO
Image by Author, Data Source: USPTO

Every week the USPTO releases the latest online Patent Gazette that lists the most recently granted utility, design, and plant patents [4]. The Patent Gazettes are available for bulk download here, https://bulkdata.uspto.gov/. This site has the gazettes from July 2002 to today.

For this project, I downloaded and processed only the design patents that have expired, so I grabbed the gazettes from July 2002 to February 2007, yielding 67,578 design patents.

An interesting aspect of design patents is the use of "broken lines" to show elements of the depicted object that are not to be considered part of the design patent [5].

"A broken line disclosure is understood to be for illustrative purposes only and forms no part of the claimed design. Structure that is not part of the claimed design, but is considered necessary to show the environment in which the design is used, may be represented in the drawing by broken lines." – USPTO

Footwear Upper USD0534338, by Manon Belley and Martin W. F. Dean, Source: USPTO
Footwear Upper USD0534338, by Manon Belley and Martin W. F. Dean, Source: USPTO

In the above example, you can see that the broken lines with the dash-dot-dot pattern are not included in the patented design, but the stitches on the top of the shoe are included. Note that all lines were included in the images that were fed into the CLIP image encoder.

Preprocessing the Design Patents

I wrote some Python code to download the bulk Patent Gazettes, unzip them, and iterate through the HTML pages looking for design patents. They’re fairly easy to find because design patents start with the prefix "USD". For example, here are the paths to an HML document and corresponding PNG image:

gazettes/1314-1/OG/html/1314-1/USD0534338-20070102.html
gazettes/1314-1/OG/html/1314-1/USD0534338-20070102.png

Inside the HTML doc, I grab the patent title and save it to a text file. I also resize and pad the image to 224×224 pixels, the size used by the CLIP image encoder. The full source code is available here.

Creating Embeddings using CLIP, Image by Author
Creating Embeddings using CLIP, Image by Author

Creating Text and Image Embeddings Using CLIP

After the text and images are preprocessed, it’s fairly easy to create embeddings to be searched. OpenAI has released pretrained models for CLIP. After downloading their model, all it took was a couple of lines of Python code to generate the embeddings. Each embedding is a list of 512 numbers. You can see the source code is here.

US Patent Office, Circa 1925, Source: US Library of Congress, Public Domain
US Patent Office, Circa 1925, Source: US Library of Congress, Public Domain

Searching for Design Patents

Now that we have the text and image embeddings for over 67 thousand design patents, running a text search is fast and easy.

Searching starts with typing in a query text, for example, "curved water faucet". The system then creates a text embedding for the query using the CLIP text encoder. You can choose one of the three methods for searching.

1. Text to text search – This compares the embedding of the query to the list of text embeddings for the design patent captions and returns the top result.

Diagram by Author
Diagram by Author

2. Text to image search – This compares the query embedding to the list of image embeddings and returns the top results.

Diagram by Author
Diagram by Author

3. Text to combined text/image search – This replicates the query embedding and compares the combined pair to a list of combined text and image embeddings and returns the top results.

Diagram by Author
Diagram by Author

Note that using both text and image embeddings for semantic search is not new. For example, you can read about recent developments in these papers, "Dialog-based Interactive Image Retrieval" by Xiaoxiao Guo et al. [6] and "Visual-Semantic Matching by Exploring High-Order Attention and Distraction" by Yongzhi Li, et al. [7].

Discussion

After experimenting with the system, I found that using the combined embeddings from both the text and image often yields the best result. The captions for the design patents are very high level, like "Men’s boot". They typically do not contain text that describes any of the key features. The image embeddings, however, will often find the "wrong" object that happens to look like the searched object. For example, it may show a candlestick if you search for a "sports trophy". But the image embeddings will capture some key aspects of the design, like the "curved water faucet" example above. Combining the two embeddings for the search is often the best of both worlds. The text embedding keeps the search limited to the scope of the query, where the image embedding helps refine the search by identifying key features.

Check out the appendix below to see some sample search results.

Future Work

It might be possible to train a Generative Adversarial Network (GAN) to generate new designs based on images from the USPTO. This would be similar to what OpenAI did with their DALL·E system, which uses a Variational Autoencoder to generate new images from text [8].

Source Code

All source code for this project is available on GitHub. The image files and labels are available on Kaggle. You can experiment with the code using this Google Colab. The sources are released under the CC BY-SA license.

Attribution-ShareAlike
Attribution-ShareAlike

Acknowledgments

I would like to thank Jennifer Lim and Oliver Strimpel for their help with this project.

References

[1] USPTO, "General information concerning patents", October 15, 2015, https://www.uspto.gov/patents/basics#heading-30

[2] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning Transferable Visual Models From Natural Language Supervision", January 5, 2021, https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf

[3] G. Quinn, IP Watchdog, "The First Design Patent", November 6, 2008, https://www.ipwatchdog.com/2008/11/06/the-first-design-patent-2/

[4] USPTO, "Official Gazette", https://www.uspto.gov/learning-and-resources/official-gazette

[5] USPTO, "Design Patent Application Guide", https://www.uspto.gov/patents/basics/types-patent-applications/design-patent-application-guide#broken

[6] X. Guo, H. Wu, Y. Cheng, S. Rennie, G. Tesauro, R. S. Feris, "Dialog-based Interactive Image Retrieval", December 20, 2018, https://arxiv.org/pdf/1805.00145.pdf

[7] Y. Li, D. Zhang, Y. Mu3, "Visual-Semantic Matching by Exploring High-Order Attention and Distraction", June 13, 2020, https://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Visual-Semantic_Matching_by_Exploring_High-Order_Attention_and_Distraction_CVPR_2020_paper.pdf

[8] A. Ramesh, M. Pavlov, G. Goh, S.t Gray, "DALL·E: Creating Images from Text", January 5, 2021, https://openai.com/blog/dall-e

Appendix – Search Results

Cocktail Shakers

Combined Search for "stainless steel cocktail shaker", Image by Author, Source: USPTO
Combined Search for "stainless steel cocktail shaker", Image by Author, Source: USPTO

Hairbrushes

Combined Search for "hairbrush with bristles", Image by Author, Source: USPTO
Combined Search for "hairbrush with bristles", Image by Author, Source: USPTO

Chaise Lounges

Combined Search for "chaise lounge", Image by Author, Source: USPTO
Combined Search for "chaise lounge", Image by Author, Source: USPTO

Related Articles