Object recognition for three-dimensional bodies

• US 9,424,461 B1
• Filed: 06/27/2013
• Issued: 08/23/2016
• Est. Priority Date: 06/27/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Yuan, Chang, Heller, Geoffrey Scott, Rybakov, Oleg, Ramaswamy, Sharadh, Thomas, Jim Oommen
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): LEMIEUX, IAN L
• Application Number:
  US13/929,672
• Time in Patent Office: 1,153 Days
• US Class Current: 1/1
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces

Abstract

Various embodiments utilize two-dimensional (“2D”) and three-dimensional (“3D”) object features for purposes such as object recognition and/or image matching. For example, a user can capture an image (e.g., still images or video) of an object and can receive information about items that are determined to match the object. For example, the image can be analyzed to detect visual features (e.g., corners, edges, etc.) of the object and the detected visual features can be combined to generate a combined visual feature vector which can be used for object recognition, image matching, or other such purposes. Other approaches utilize the image to generate a 3D model of the object represented in the image, which can be used to determine at least one object or types of objects that match the object represented in the image.

Claims

1. A computer device, comprising:

at least one processor;

a camera; and

memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive an image acquired by the camera, the image including a representation of an object;

analyze the image to identify a plurality of feature points;

generate, based at least in part upon the plurality of feature points, a plurality of visual feature vectors representative of the object;

determine a distinctive score for each of the plurality of visual feature vectors, the distinctive score based at least in part on indispensable information that is useful in identifying the object, wherein the indispensable information represents frequently occurring features;

select a subset of the plurality of visual feature vectors having a distinctive score equal or above a distinctive score threshold;

generate a compact combined visual feature vector using the selected subset of the plurality of visual feature vectors;

compare the compact combined visual feature vector to one or more stored vectors of a set of stored vectors, each of the set of stored vectors corresponding to a respective type of object;

determine a matching stored vector having a respective similarity score that at least meets a matching threshold; and

identify, based at least in part on the matching stored vector, at least one respective type of object represented in the image.

2. The computing device of claim 1, wherein the instructions, when executed to generate a compact combined visual feature vector include combining related vector representations.

3. The computing device of claim 1, wherein the instruction, when executed to compare the compact combined visual feature vector, further cause the computing device to:

analyze differences between the compact combined visual feature vector and the one or more stored vectors of a set of stored vectors to identify at least one representative type of object, and wherein differences between the compact combined visual feature vector and the one or more stored vectors are less than a determined matching threshold.

4. A computer implemented method, comprising:

receiving an image acquired by a camera of a computing device, the image including a representation of an object;

determining a plurality of visual feature vectors representative of the object;

generating a combined visual feature vector using at least a first subset of the plurality of visual feature vectors;

determining a distinctive score for each visual feature vector of the first subset of the plurality of visual feature vectors, the distinctive score based at least in part on indispensable information that is useful in identifying the object, wherein the indispensable information represents frequently occurring features; and

selecting a second subset of the plurality of visual feature vectors having the distinctive score above or equal to a distinctive score threshold; and

generating a compact combined visual feature vector using the second subset of the plurality of visual feature vectors, wherein the compact combined visual feature vector is configured to identify at least one object type associated with the representation of the object.

5. The computer implemented method of claim 4, wherein selecting the second subset of the plurality of visual feature vectors includes at least one of removing a subset of the plurality of visual feature vectors, or rearranging an order of a subset of the plurality of visual feature vectors.

6. The computer implemented method of claim 4, further comprising:

receiving a visual search query, wherein in response to the visual search query further performing;

comparing the compact combined visual feature vector to one or more stored vectors of a set of stored vectors, each of the set of stored vectors corresponding to a respective type of object;

determining a matching stored vector having a respective similarity score that at least meets a matching threshold; and

identifying, based at least in part on the matching, at least one respective type of object represented in the image, the at least one respective type of object being associated with identifying information.

7. The computer implemented method of claim 6, further comprising:

providing the identifying information to the computing device, wherein the identifying information includes at least one of a name of the at least one respective type of object, a description of physical characteristics associated with the at least one respective type of object, pricing information associated with the at least one respective type of object, history information associated with the at least one respective type of object, or a reference to one or more resources for information associated with the at least one respective type of object.

8. The computer implemented method of claim 6, wherein comparing the compact combined visual feature vector to one or more stored vectors further includes:

analyzing differences between the compact combined visual feature vector and the one or more stored vectors of a set of stored vectors to identify at least one representative type of object, wherein differences between the compact combined visual feature vector and the one or more stored compact vectors are less than a determined matching threshold.

9. The computer implemented method of claim 6, wherein comparing the compact combined visual feature vector to one or more stored vectors further includes:

identifying a subset of feature vectors of the compact combined visual feature vector;

matching the identified subset of feature vectors to one or more stored vectors of a set of stored features, each of the set of stored vectors corresponding to a respective type of object;

generating, based at least in part on the matching, a list of respective types of objects; and

identifying corresponding respective types of objects between lists of types of objects.

10. The computer implemented method of claim 4, wherein the plurality of visual feature vectors includes at least one of an image descriptor or a three-dimensional model.

11. The computer implemented method of claim 4, wherein generating the compact combined visual feature vector includes combining at least a subset of related visual feature vectors.

12. The computer implemented method of claim 4, wherein determining the plurality of visual feature vectors includes using at least one of a Moravec corner detection algorithm, a Harris corner detection algorithm, a FAST corner detection algorithm a scale-invariant feature transform (SIFT) algorithm, a speeded up robust feature (SURF) algorithm, a gradient location and orientation histogram algorithm, a rotation-invariant feature transform (RIFT) algorithm, a Canny algorithm, or a local texture descriptor algorithm.

13. The computer implemented method of claim 4, further comprising adjusting an order of the second subset of the plurality of visual feature vectors based at least in part on the distinctive score associated with each visual feature vectors of the second subset of the plurality of visual feature vectors.

14. A computing device, comprising:

a processor;

a camera; and

memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive an image captured by the camera, the image including a representation of an object;

determine a plurality of visual feature vectors representative of the object;

generate a combined visual feature vector using at least a first subset of the plurality of visual feature vectors;

determine a distinctive score associated with the first subset of the plurality of visual feature vectors, the distinctive score based at least in part on indispensable information that is useful in identifying the object, wherein the indispensable information represents frequently occurring features;

select a second subset of the plurality of visual feature vectors having the distinctive score above or equal to a distinctive score threshold; and

generate a compact combined visual feature vector using the second subset of the plurality of visual feature vectors, wherein the compact combined visual feature vector is configured to identify at least one object type associated with the representation of the object.

15. The computing device of claim 14, wherein selecting the second subset of the plurality of visual feature vectors includes at least one of removing a subset of the plurality of visual feature vectors, or rearranging an order of a subset of the plurality of visual feature vectors.

16. The computing device of claim 14, wherein the instruction, when executed, further causes the computing device to:

receive a visual search query, wherein in response to the visual search query further performing;

compare the compact combined visual feature vector to one or more stored vectors of a set of stored vectors, each of the set of stored vectors corresponding to a respective type of object;

determine a matching stored vector having a respective similarity score that at least meets a matching threshold; and

identify, based at least in part on the matching, at least one respective type of object represented in the image, the at least one respective type of object being associated with identifying information.

17. The computing device of claim 16, wherein the instruction, when executed, further causes the computing device to:

provide the identifying information to the computing device, wherein the identifying information includes at least one of a name of the at least one respective type of object, a description of physical characteristics associated with the at least one respective type of object, pricing information associated with the at least one respective type of object, history information associated with the at least one respective type of object, or a reference to one or more resources for information associated with the at least one respective type of object.