Identifying items in images using regions-of-interest

• US 9,953,242 B1
• Filed: 12/21/2015
• Issued: 04/24/2018
• Est. Priority Date: 12/21/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Tyagi, Ambrish, Fotland, David Allen
• Primary Examiner(s): Wu, Jingge
• Application Number:
  US14/976,850
• Time in Patent Office: 855 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/529 from texture
• CPC Class Codes: G06V 10/22 by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 10/759 Region-based matching
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

The techniques described herein may identify images that likely depict one or more items by comparing features of the items to features of different regions-of-interest (ROIs) of the images. When a user requests to identify images that depict a particular item, the techniques may determine a region-of-interest (ROI) size based on the size of the requested item. The techniques may then search multiple images using the ROI size.

Claims

1. A method comprising:

storing first image data that represents a user wearing at least a first item of clothing;

storing second image data that represents the user wearing at least a second item of clothing;

receiving a request to identify at least one image that depicts an object represented in third image data, the object comprising the first item of clothing;

identifying first pixel data that represents the object in the third image;

determining an object size of the object in the third image data based at least in part on the first pixel data;

determining a first depth of the object represented in the third image data;

determining, using a first depth map associated with the first image data, a second depth of the user represented in the first image data;

determining, using a second depth map associated with the second image data, a third depth of the user represented in the second image data;

determining a first region-of-interest (ROI) size based at least in part on the object size of the object in the third image, the first depth, and the second depth, and the first ROI size representing a first size of an image region to search;

analyzing a first portion of the first pixel data corresponding to a first region of the first image data, the first region of the first image data having the first ROI size;

analyzing a second portion of the first pixel data corresponding to a second region of the first image data, the second region of the first image data having the first ROI size;

determining a second ROI size based at least in part on the object size of the object in the third image data, the first depth, and the third depth, and the second ROI size representing a second size of an image region to search;

analyzing a third portion of the first pixel data corresponding to a third region of the second image data, the third region of the second image data having the second ROI size;

analyzing a fourth portion of the first pixel data corresponding to a fourth region of the second image data, the fourth region of the second image data having the second ROI size;

determining the first region of the first image data depicts the object; and

outputting an indication that the first image data depicts the object.

2. The method as recited in claim 1, wherein:

the determining the first ROI size comprises determining the first ROI size as the function of the product of the object size of the object in the third image data and the ratio of the first depth to the second depth; and

the determining the second ROI size comprises determining the second ROI size as the function of the product of the size of the object in the third image data and the ratio of the first depth to the third depth.

3. The method as recited in claim 1, wherein the analyzing the first portion of the first pixel data corresponding to the first region of the first image data comprises:

computing a first color histogram of a portion of the third image data that represents the object;

computing a second color histogram of the first region of the first image data;

analyzing the first color histogram with respect to the second color histogram;

determining that the first color histogram and the second color histogram have a color similarity this is greater than a threshold color similarity; and

at least partly in response to determining that the first color histogram and the second color histogram have the color similarity this is greater than the threshold color similarity, analyzing the first portion of the first pixel data corresponding to the first region of the first image data using a pre-trained convolutional neural network.

4. The method as recited in claim 1, wherein:

the first region of the first image data and the second region of the first image data at least partly overlap; and

the third region of the second image data and the fourth region of the second image data at least partly overlap.

5. The method as recited in claim 1, further comprising:

defining a first bounding box of the first image data, the first bounding box including pixels corresponding to the user and excluding at least a portion of pixels corresponding to a background of the first image data; and

defining a second bounding box of the second image data, the second bounding box including pixels corresponding to the user and excluding at least a portion of pixels corresponding to a background of the second image data;

wherein the first region of the first image data and the second region of the first image data reside within the first bounding box, and the third region of the second image data and the fourth region of the second image data reside within the second bounding box.

6. A method comprising:

receiving a request to identify at least one image that includes a first object;

identifying first pixel data that represents the first object in first image data;

determining an object size of the first object in the first image data based at least in part on the first pixel data;

determining a depth of at least a portion of a second object represented in the first image data;

calculating a region-of-interest (ROI) size based at least in part on;

the object size of the first object in the first image data; and

the depth of the second object in the first image data;

analyzing a first portion of the first pixel data corresponding to a first region of the first image data, the first region of the first image data having the ROI size;

analyzing a second portion of the first pixel data corresponding to a second region of the first image data, the second region of the first image data having the ROI size;

determining that the first region depicts the first object; and

outputting an indication that the first image data depicts the first object.

7. The method as recited in claim 6, further comprising:

receiving a request to identify at least one image that includes a particular item of clothing;

determining a depth of a least a portion of a user in the first image data; and

determining that the user is wearing the particular item of clothing in at least one of the first region or the second region.

8. The method as recited in claim 6, wherein:

the receiving the request comprises receiving a request to identify at least one image that includes a first object represented in second image data;

the determining the depth comprises determining a first depth;

and further comprising;

determining a second object size of the first object in the second image data; and

determining a second depth of the first object in the second image data; and

wherein the calculating the ROI size comprises calculating the ROI size based at least in part on the first depth, the size of the first object in the second image data, and the second depth.

9. The method as recited in claim 8, wherein the calculating the ROI size comprises calculating the ROI size as a function of a product of the second object size of the first object in the second image data and a ratio of the second depth to the first depth.

10. The method as recited in claim 6, wherein analyzing the first portion of the first pixel data corresponding to the first region comprises:

computing a first color histogram of the first object;

computing a second color histogram of at least a portion of the first region;

analyzing the first color histogram with respect to the second color histogram; and

determining a color similarity between the first color histogram and the second color histogram.

11. The method as recited in claim 10, wherein the analyzing the first portion of the first pixel data corresponding to the first region further comprises:

determining that the color similarity is greater than a threshold color similarity; and

analyzing the first portion of the first pixel data corresponding to the first region using a pre-trained classifier.

12. The method as recited in claim 6, wherein:

the determining the depth comprises determining a first depth; and

the calculating the ROI size comprises calculating a first ROI size;

and further comprising;

determining a second depth of a least a portion of a third object in second image data;

calculating a second ROI size based at least in part on the second depth of the third object in the second image data;

analyzing a third portion of the first pixel data corresponding to a third region of the second image data, the third region of the second image data having the second ROI size; and

analyzing a fourth portion of the first pixel data corresponding to a fourth region of the second image data, the fourth region of the second image data having the second ROI size.

13. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising;

receiving a request to identify at least one image that includes a first object;

identifying first pixel data that represents the first object in first image data;

determining an object size of the first object in the first image data based at least in part on the first pixel data;

determining a depth of at least a portion of a second object represented in the first image data;

calculating a region-of-interest (ROI) size based at least in part on;

the object size of the first object in the first image data;

the depth of the second object in the first image data;

analyzing a first portion of the first pixel data corresponding to a first region of the first image data, the first region of the first image data having the ROI size;

analyzing a second portion of the first pixel data corresponding to a second region of the first image data, the second region of the first image data having the ROI size;

determining at least one of that the first region represents the first object or that the second region depicts the first object; and

outputting an indication that the first image data represents the first object.

14. The system as recited in claim 13, wherein:

the receiving the request comprises receiving a request to identify at least one image that includes a particular item of clothing;

the determining the depth comprises determining a depth of a least a portion of a user in the first image data; and

the determining that the at least one of the first region represents the first object or that the second region represents the first object comprises determining that the user is wearing the particular item of clothing in at least one of the first region or the second region.

15. The system as recited in claim 13, wherein:

the receiving the request comprises receiving a request to identify at least one image that includes a first object represented in second image data;

the determining the depth comprises determining a first depth;

and the acts further comprising;

determining a second object size of the first object in the second image data; and

determining a second depth of the first object in the second image data; and

wherein the calculating the ROI size comprises calculating the ROI size based at least in part on the first depth, the size of the first object in the second image data, and the second depth.

16. The system as recited in claim 15, wherein the calculating the ROI size comprises calculating the ROI size as a function of a product of the second object size of the first object in the second image data and a ratio of the second depth to the first depth.

17. The system as recited in claim 13, wherein analyzing the first portion of the first pixel data corresponding to the first region comprises:

computing a first color histogram of the first object;

computing a second color histogram of at least a portion of the first region;

analyzing the first color histogram with respect to the second color histogram; and

determining a color similarity between the first color histogram and the second color histogram.

18. The system as recited in claim 17, wherein the analyzing the first portion of the first pixel data corresponding to the first region further comprises:

determining that the color similarity is greater than a threshold color similarity; and

analyzing the first portion of the first pixel data corresponding to the first region using a pre-trained classifier.

19. The system as recited in claim 13, wherein:

the determining the depth comprises determining a first depth; and

the calculating the ROI size comprises calculating a first ROI size;

and the acts further comprising;

determining a second depth of a least a portion of a third object in second image data;

calculating a second ROI size based at least in part on the second depth of the third object in the second image data;

analyzing a third portion of the first pixel data corresponding to a third region of the second image data, the third region of the second image data having the second ROI size; and

analyzing a fourth portion of the first pixel data corresponding to a fourth region of the second image data, the fourth region of the second image data having the second ROI size.

20. The system as recited in claim 13, wherein the first region of the first image data and the second region of the first image data at least partly overlap.