Hand-drawn sketch recognition

• US 9,147,125 B2
• Filed: 05/03/2013
• Issued: 09/29/2015
• Est. Priority Date: 05/03/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wang, Changhu, Zhang, Lei, Sun, Zhenbang
• Primary Examiner(s): Liu, Li
• Application Number:
  US13/886,904
• Publication Number:
  US 20140328544A1
• Time in Patent Office: 879 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5854 using shape and object relationship

Abstract

Some examples of a sketch-based image recognition system may generate a model for identifying a subject of a sketch. The model is formed from a plurality of images having visual features similar to the visual features of the sketch. The model may include object topics representative of categories which may correspond to the subject of the sketch and shape topics representative of the visual features of the sketch.

Claims

1. A computing device comprising:

one or more input interfaces for receiving a sketch;

one or more processors; and

a computer-readable storage media storing instructions, which when executed by the one or more processors, cause the one or more processors to generate a model for identifying a subject of the sketch;

performing an image-based search of a collection of images to identify a set of images with shape features similar to those of the sketch;

filtering the set of images into a set of filtered images by removing images that have more than one subject, each image of the set of filtered images associated with at least one text-based label;

deriving a shape feature for each image of the set of filtered images;

generating a plurality of shape topics based on the shape features of the set of filtered images;

generating a plurality of object topics based on the shape topics and associating each of the object topics with at least one of the plurality of shape topics;

deriving a word distribution for each of the object topics from the text-based labels;

identifying a word descriptive of the subject of the sketch from at least one of the word distributions; and

applying the word as a tag to the sketch.

2. The computing device of claim 1, further comprising calculating a probability for each word associated with the word distributions and selecting a word from the word distribution as the tag, the word having a highest resulting probability.

3. The computing device of claim 1, wherein images within the collection of images are clipart images.

4. The computing device of claim 1, wherein the collection of images is accessed via one or more communication interfaces.

5. The computing device of claim 1, wherein the sketch is added to the collection of images.

6. The computing device of claim 1, further comprising performing one or more predefined actions based on the identity of the sketch.

7. A method comprising:

receiving, from a computing device over a network, a sketch;

searching a database of images using the sketch as a query to identify a set of images, the set of images having shape features similar to at least one shape feature of the sketch;

generating a model based on the set of images, the model to aid in identifying a subject of the sketch;

determining, based at least in part on text-based labels associated with individual images from the set of images in the model, the subject of the sketch; and

tagging the sketch with the subject.

8. The method of claim 7, wherein the searching is performed by an image-based search engine.

9. The method of claim 7, wherein the model includes object topics and shape topics.

10. The method of claim 9, wherein the object topics represent categories of subjects which may be associated with the sketch.

11. The method of claim 9, wherein the shape topics represent shape features associated with the sketch.

12. The method of claim 9, wherein at least one image of the set of images is associated with one of the shape topics based at least in part on at least one shape feature of the at least one image.

13. The method of claim 9, wherein:

at least one image from the set of images is associated with a shape topic of the shape topics;

the shape topic is associated with an object topic of the object topics; and

the object topic includes at least one word descriptive of the object topic based at least in part on a text-based label associated with the at least one image.

14. The method of claim 9, wherein the subject of the sketch is determined by ranking probabilities, the probabilities calculated for at least first and second object topics of the object topics, wherein the probabilities are based at least in part on at least one word associated with the first and second object topics and one or more shape topics associated with the first and second object topics.

20. The method of claim 7, wherein:

the model includes object topics and shape topics,

at least one image from the set of images is associated with at least one of the shape topics, the at least one of the shape topics being associated with a text-based label associated with the at least one image, and

individual words of the text-based label are ranked based at least in part on a probability that the individual words are descriptive of the subject of the sketch.

15. A computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to generate a model for identifying a subject of a sketch, the model including:

at least one object topic, the at least one object topic representative of a category which may correspond to the subject of the sketch;

at least one shape topic associated with the at least one object topic, the at least one shape topic representative of a shape feature of the sketch; and

at least one image associated with the at least one shape topic, wherein the at least one shape topic is associated with a word distribution formed from a textual descriptor associated with the at least one image, and wherein individual words of the word distribution are ranked based at least in part on a probability that the individual words are descriptive of the subject of the sketch.

16. The computer-readable storage media of claim 15, wherein the at least one image includes the shape feature associated with the at least one shape topic and the textual descriptor.

17. The computer-readable storage media of claim 16, wherein the at least one image is received from an image-based search engine as a result of a search performed by the image-based search engine using the sketch as a query.

18. The computer-readable storage media of claim 15, wherein the at least one image is associated with the at least one shape topic based at least in part on a probability that the at least one image has the shape feature of the at least one shape topic.

19. The computer-readable storage media of claim 15, wherein the at least one shape topic is associated with the at least one object topic based at least in part on a probability that the shape feature of the at least one shape topic belongs to the category of the at least one object topic.