Object retrieval

• US 10,013,633 B1
• Filed: 03/08/2017
• Issued: 07/03/2018
• Est. Priority Date: 12/23/2014
• Status: Expired due to Fees

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Manmatha, Raghavan, Chuang, Wei-Hong
• Primary Examiner(s): Potts, Ryan P
• Application Number:
  US15/453,726
• Time in Patent Office: 482 Days
• Field of Search: None
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2210/16 Cloth
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/421 by analysing segments intersecting the pattern
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

Various approaches enable a user to capture image information (e.g., still images or video) about an object of interest such as the sole of a shoe or other piece of footwear (e.g., a sandal) and receive information about items that are determined to match footwear based at least in part on the image information. For example, an image analyze service or other similar service can analyze the images to determine a type of shoe included within the images based at least in part on patterns of other distinguishing features of the sole of the shoe. The image analysis service can aggregate the results and can provide information about the results as a set of matches or results to be displayed to a user in response to a visual search query. The information can include, for example, descriptions, contact information, availability, location data, pricing information, and other such information.

Claims

1. A computing device, comprising:

a camera configured to capture an image;

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive the image acquired by the camera, the image including a representation of a footwear sole;

analyze the image to determine at least one portion of that image that includes a view of at least a portion of a surface pattern of the footwear sole, the portion of the surface pattern of the footwear sole including repeated surface pattern markings separated by spacing, at least a first portion of the repeated surface pattern markings being located in a toe portion of the footwear sole, a second portion of the repeated surface pattern markings being located in a heel portion of the footwear sole, and a distinguishing feature separate from the surface pattern markings on the footwear sole including at least one of a logo or text;

determine feature information representative of at least a portion of the surface pattern or markings; and

return matching footwear information based at least in part on the feature information.

2. The computing device of claim 1, wherein the instructions when executed to return matching footwear information further cause the computing device to:

compare the feature information to stored feature information representative of soles of footwear to determine a shoe type corresponding to the representation of the footwear sole.

3. The computing device of claim 2, wherein the instructions when executed to return matching footwear information further cause the computing device to:

determine matching stored feature information with a respective similarity score that at least meets a matching threshold.

4. The computing device of claim 3, wherein the instructions when executed to return matching footwear information further cause the computing device to:

identify, based at least in part on the matching feature information, at least one respective type of footwear represented in the image.

5. The computing device of claim 4, wherein the instructions when executed to return matching footwear information further cause the computing device to:

provide, for display on the computing device, at least one of a description, contact information, availability, location data, or pricing information of the respective type of footwear.

6. A computing device, comprising:

a camera configured to capture an image;

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the computing device to;

receive the image acquired by the camera, the image including a representation of a surface pattern on a portion of a footwear sole, wherein the surface pattern comprises repeated markings separated by spacing, at least a first portion of the repeated markings being located in a toe portion of the footwear sole, a second portion of the repeated markings being located in a heel portion of the footwear sole, and a distinguishing feature separate from the repeated markings on the footwear sole including at least one of a logo or text;

analyze the image to determine at least one feature representative of a portion of the surface pattern;

compare the at least one feature to stored information to determine a type of footwear represented in the image; and

return matching footwear information based at least in part on the type of footwear represented in the image.

7. The computing device of claim 6, wherein the instructions when executed to return matching footwear information further cause the computing device to:

identify, based at least in part on matching feature information to stored feature information, at least one respective type of footwear represented in the image.

8. The computing device of claim 7, wherein the instructions when executed to return matching footwear information further cause the computing device to:

provide on a display of the computing device, in response to receiving the image, the identified at least one respective type of footwear for a user action.

9. The computing device of claim 6, wherein the instructions when executed to analyze the image further cause the computing device to:

generate one or more codewords based at least in part on the analyzed image.

10. The computing device of claim 9, wherein the instructions when executed to analyze the image further cause the computing device to:

generate a codebook based at least upon the one or more codewords.

11. The computing device of claim 9, wherein the one or more codewords represent a particular appearance of the surface pattern on a sole of the footwear.

12. The computing device of claim 6, wherein the instructions when executed to analyze the image further cause the computing device to:

analyze one or more intensity gradients of a region in the image to identify one or more potential edge locations.

13. The computing device of claim 6, wherein the instructions when executed to analyze the image further cause the computing device to:

analyze one or more character portions using a character recognition algorithm to recognize at least one character associated with the one or more character portions.

14. The computing device of claim 6, wherein the surface pattern includes a spacing between different regions of a sole of the footwear, an arrangement of the repeated markings, a distance between the markings, or a number of the markings.

15. The computing device of claim 6, wherein a sole of the footwear further includes at least one of colors or art.

16. A method, comprising:

receiving an image acquired by a camera of a computing device, the image including a representation of a surface pattern on a portion of a footwear sole, wherein the surface pattern comprises repeated markings separated by spacing, at least a first portion of the repeated markings being located in a toe portion of the footwear sole, a second portion of the repeated markings being located in a heel portion of the footwear sole, and a distinguishing feature separate from the repeated markings on the footwear sole including at least one of a logo or text;

analyzing the image with an image analysis service associated with a content provider to determine at least one feature representative of a portion of the surface pattern;

generating one or more codewords using at least one of an edge detection algorithm, a feature detection algorithm, or a gradient detection algorithm;

comparing the at least one feature to stored information associated with the content provider to determine a type of shoe represented in the image, wherein the at least one feature is mapped to a particular codeword of the one or more codewords; and

returning matching footwear information based at least in part on the type of shoe represented in the image.

17. The method of claim 16, further comprising:

identifying, based at least in part on matching feature information to stored feature information, at least one respective type of footwear represented in the image.

18. The method of claim 17, further comprising:

providing on a display of the computing device, in response to receiving the image, the identified at least one respective type of footwear for a user action.

19. The method of claim 16, wherein analyzing the image further comprises:

generating the one or more codewords based at least in part on the analyzed image.

20. The method of claim 19, wherein generating the one or more codewords further comprises:

generating the one or more codewords using at least one of a Harris corner detection algorithm, a FAST corner detection algorithm a scale-invariant feature transform (SIFT) algorithm, a speeded up robust feature (SURF) algorithm, an accumulated signal gradient (ASG) algorithm, a gradient location and orientation histogram algorithm, or a rotation-invariant feature transform (RIFT) algorithm.