THREE-DIMENSIONAL VISUAL PHRASES FOR OBJECT RECOGNITION

• US 20140029856A1
• Filed: 07/30/2012
• Published: 01/30/2014
• Est. Priority Date: 07/30/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cai, Rui, Li, Zhiwei, Zhang, Lei, Hao, Qiang
• Granted Patent: US 8,983,201 B2
• Time in Patent Office: Days
• US Class Current: 382/195
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations

Abstract

The techniques discussed herein discover three-dimensional (3-D) visual phrases for an object based on a 3-D model of the object. The techniques then describe the 3-D visual phrases. Once described, the techniques use the 3-D visual phrases to detect the object in an image (e.g., object recognition).

Claims

1. A method comprising:

under control of a processor configured with computer-executable instructions,

receiving a collection of images each containing an object;

constructing a three-dimensional (3-D) model of the object, the 3-D model including a plurality of points;

selecting at least a subset of the plurality of points;

generating one or more 3-D visual phrases based on the subset of the plurality of points; and

using the one or more 3-D visual phrases to detect the object in an unclassified image.

2. The method recited in claim 1, wherein a 3-D visual phrase is a triangular facet on a surface of the 3-D model.

3. The method recited in claim 1, further comprising describing each of the one or more 3-D visual phrases by characterizing a visual appearance of each point in the 3-D visual phrase.

4. The method recited in claim 1, further comprising describing each of the one or more 3-D visual phrases by characterizing a geometric structure of points in the 3-D visual phrase.

5. The method recited in claim 1, wherein using the one or more 3-D visual phrases to detect the object in the unclassified image comprises matching visual appearances of one or more points in the one or more 3-D visual phrases with scale-invariant feature transform (SIFT) features extracted from the unclassified image.

6. The method recited in claim 1, wherein using the one or more 3-D visual phrases to detect the object in the unclassified image comprises comparing a geometric structure of each 3-D visual phrase to a cyclic order of features extracted from the unclassified image.

7. The method recited in claim 1, wherein the collection of images is a set of training images known to include the object.

8. The method recited in claim 1, wherein the selecting the subset of the plurality of points is performed in accordance with a sampling rate.

9. The method recited in claim 1, wherein the object is a landmark and the collection of images are received from multiple different locations on a network.

10. The method recited in claim 1, further comprising storing the one or more 3-D visual phrases for object recognition.

11. One or more computer storage media storing computer executable instructions that, when executed, perform operations comprising:

storing one or more sets of three-dimensional (3-D) visual phrases, each set of 3-D visual phrases being associated with an object;

receiving an indication to perform object recognition for one or more images;

performing object recognition on the one or more images using the one or more sets of 3-D visual phrases; and

categorizing a portion of the one or more images based on an object recognized using the one or more sets of 3-D visual phrases.

12. The one or more computer storage media recited in claim 11, wherein the indication is based on an image-based search query received from a client device.

13. The one or more computer storage media recited in claim 12, further performing an operation comprising providing image-based search results that include one or more of the categorized images associated with the image-based search query.

14. A system comprising:

one or more processors;

one or more computer memories, coupled to the one or more processors and storing;

a image access module, operable by the one or more processors, that accesses a plurality of images that each comprise an object;

a three-dimensional (3-D) visual phrase discoverer module, operable by the one or more processors, that discovers one or more 3-D visual phrases for the object;

a 3-D visual phrase description module, operable by the one or more processors, that characterizes the one or more 3-D visual phrases; and

an object detection module, operable by the one or more processors, that uses the one or more characterized 3-D visual phrases to recognize the object in an image.

15. The system as recited in claim 14, wherein the three-dimensional (3-D) visual phrase discoverer module discovers the one or more 3-D visual phrases by constructing a 3-D point cloud for the object using the plurality of the images, and each 3-D visual phrase is a triangular facet on a surface of the 3-D point cloud.

16. The system as recited in claim 14, wherein the 3-D visual phrase description module characterizes each of the one or more 3-D visual phrases by describing a visual appearance for each point in the 3-D visual phrase and describing a geometric structure of all the points in the 3-D visual phrase.

17. The system as recited in claim 14, wherein the object detection module uses the one or more characterized 3-D visual phrases to recognize the object in the image by matching visual appearances of one or more points in the one or more characterized 3-D visual phrases with scale-invariant feature transform (SIFT) features extracted from the image.

18. The system as recited in claim 14 wherein the object detection module uses the one or more characterized 3-D visual phrases to recognize the object in the image by comparing a geometric structure of each 3-D visual phrase to a cyclic order of features extracted from the image.

19. The system as recited in claim 14, wherein the plurality of images are known to contain the object and the image is not known to contain the object.

20. The system as recited in claim 19, wherein at least a portion of the plurality of images are photographs of a landmark accessed via the Internet.