Hand-drawn sketch recognition

• US 9,870,516 B2
• Filed: 09/08/2015
• Issued: 01/16/2018
• Est. Priority Date: 05/03/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wang, Changhu, Zhang, Lei, Sun, Zhenbang
• Primary Examiner(s): Liu, Li
• Application Number:
  US14/847,834
• Publication Number:
  US 20150379375A1
• Time in Patent Office: 861 Days
• CPC Class Codes: G06F 16/5854 using shape and object relationship

Abstract

Some examples of a sketch-based image recognition system may generate a model for identifying a subject of a sketch. The model is formed from a plurality of images having visual features similar to the visual features of the sketch. The model may include object topics representative of categories which may correspond to the subject of the sketch and shape topics representative of the visual features of the sketch.

Claims

1. A method comprising:

receiving, from a computing device over a network, a sketch;

searching a database of images using the sketch as a query to identify a set of images, an individual image of the set of images including a shape feature similar to at least one shape feature of the sketch;

determining, based at least in part on the set of images, at least one object topic; and

generating, based at least in part on the set of images, a model for identifying a subject of the sketch, the model including;

the at least one object topic, the at least one object topic being representative of a category that is descriptive of the subject of the sketch; and

at least one shape topic associated with the at least one object topic, the at least one shape topic representative of a shape feature of the sketch.

2. The method of claim 1, wherein the individual image of the set of images further includes at least one text-based label, and wherein the method further comprises:

generating a word distribution for the at least one object topic from text-based labels of at least a portion of the set of images; and

identifying a word descriptive of the subject of the sketch from the word distribution.

3. The method of claim 2, wherein the word distribution includes at least a first word and a second word, and wherein the method further comprises:

determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject of the sketch;

determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject of the sketch; and

generating a ranking for the first word and the second word based at least in part on the first rank and the second rank,

and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking.

4. The method of claim 1, wherein the at least one object topic includes a first object topic and a second object topic, and wherein the method further comprises:

determining a first probability that a shape topic of the at least one shape topic belongs to the first object topic;

determining a second probability that the shape topic belongs to the second object topic; and

associating the shape topic with at least one of the first object topic or the second object topic based at least in part on the first probability and the second probability.

5. The method of claim 4, wherein:

the first probability is greater than the second probability; and

associating the shape topic with the at least one of the first object topic or the second object topic comprises associating the shape topic with the first object topic based at least in part on the first probability for the first shape topic being greater than the second probability for the second object topic.

6. The method of claim 1, further comprising:

filtering the set of images into a set of filtered images by removing images that include more than one subject,

and wherein generating the model based at least in part on the set of images comprises generating the model based at least in part on the set of filtered images.

7. The method of claim 1, further comprising:

identifying, for at least an image of the set of images, a shape feature of the image; and

associating the image with a shape topic of the at least one shape topic based at least in part on the shape feature of the image.

20. The method of claim 1, further comprising identifying a word descriptive of the subject of the sketch using the model.

8. A computing device comprising:

one or more input interfaces for receiving a sketch;

one or more processors; and

a computer-readable storage media storing instructions, which when executed by the one or more processors, cause the one or more processors to perform operations comprising;

performing an image-based search of a database of images using the sketch as a search query;

identifying a set of images, an individual image of the set of images including a shape feature similar to a shape feature of the sketch;

determining an object topic based at least in part on the set of images; and

generating, based at least in part on the set of images, a model for identifying a subject associated with the sketch, the model including;

the object topic, the object topic representative of a category for the subject that is associated with the sketch; and

a shape topic associated with the object topic, the shape topic representative of the shape feature of the sketch.

9. The computing device of claim 8, wherein the individual image of the set of images is associated with at least one text-based label, and wherein the operations further comprise:

generating a word distribution for the object topic from text-based labels associated with at least a portion of the set of images; and

identifying a word descriptive of the subject associated with the sketch from the word distribution.

10. The computing device of claim 9, wherein the word distribution includes at least a first word and a second word, and wherein the operations further comprise:

determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject associated with the sketch;

determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject associated with the sketch; and

generating a ranking for the first word and the second word based at least in part on the first rank and the second rank,

and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking.

11. The computing device of claim 8, wherein the model further includes an additional object topic, and wherein the operations further comprise:

determining a first probability that the shape topic is associated with the object topic;

determining a second probability that the shape topic is associated with the additional object topic; and

associating the shape topic with the object topic based at least in part on the first probability and the second probability.

12. The computing device of claim 8, the operations further comprising:

filtering the set of images into a set of filtered images by removing images that include more than one subject,

and wherein generating the model based at least in part on the set of images comprises generating the model based at least in part on the set of filtered images.

13. The computing device of claim 8, the operations further comprising:

identifying, for at least an image of the set of images, a shape feature of the image; and

associating the image with the shape topic based at least in part on the shape feature of the image.

14. A computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

searching a database of images using a sketch as a query to identify a set of images, an individual image of the set of images including a shape feature similar to at least one shape feature of the sketch;

determining, based at least in part on the set of images, at least one object topic; and

generating, based at least in part on the set of images, a model for identifying a subject of the sketch, the model including;

the at least one object topic, the at least one object topic being representative of a category that is descriptive of the subject of the sketch; and

at least one shape topic associated with the at least one object topic, the at least one shape topic representative of a shape feature of the sketch.

15. The computer-readable storage media of claim 14, wherein the individual image of the set of images further includes at least one text-based label, and wherein the operations further comprise:

generating a word distribution for the at least one object topic from text-based labels of at least a portion the set of images; and

identifying a word descriptive of the subject of the sketch from the word distribution.

16. The computer-readable storage media of claim 15, wherein the word distribution includes at least a first word and a second word, and wherein the operations further comprise:

determining a first rank for the first word based at least in part on a first probability that the first word is descriptive of the subject of the sketch;

determining a second rank for the second word based at least in part on a second probability that the second word is descriptive of the subject of the sketch; and

generating a ranking for the first word and the second word based at least in part on the first rank and the second rank,

and wherein identifying the word descriptive of the subject of the sketch is based at least in part on the ranking.

17. The computer-readable storage media of claim 14, wherein the at least one object topic includes a first object topic and a second object topic, and wherein the operations further comprise:

determining a first probability that a shape topic of the at least one shape topic belongs to the first object topic;

determining a second probability that the shape topic belongs to the second object topic; and

associating the shape topic with at least one of the first object topic or the second object topic based at least in part on the first probability and the second probability.

18. The computer-readable storage media of claim 17, wherein:

the first probability is greater than the second probability; and

associating the shape topic with the at least one of the first object topic or the second object topic comprises associating the shape topic with the first object topic based at least in part on the first probability for the first shape topic being greater than the second probability for the second object topic.

19. The computer-readable storage media of claim 14, the operations further comprising:

identifying, for at least an image of the set of images, a shape feature of the image; and

associating the image with a shape topic of the at least one shape topic based at least in part on the shape feature of the image.