Identifying items in images using regions-of-interest

• US 10,007,860 B1
• Filed: 12/21/2015
• Issued: 06/26/2018
• Est. Priority Date: 12/21/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Fotland, David Allen, Tyagi, Ambrish
• Primary Examiner(s): Bhatnagar, Anand
• Application Number:
  US14/976,696
• Time in Patent Office: 918 Days
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

The techniques described herein may identify images that likely depict one or more items by comparing features of the items to features of different regions-of-interest (ROIs) of the images. For instance, some of the images may include a user, and the techniques may define multiple regions within the image corresponding to different portions of the user. The techniques may then use a trained convolutional neural network or any other type of trained classifier to determine, for each region of the image, whether the region depicts a particular item. If so, the techniques may designate the corresponding image as depicting the item and may output an indication that the image depicts the item. The techniques may perform this process for multiple images, outputting an indication of each image deemed to depict the particular item.

Claims

1. A method comprising:

storing first image data that depicts a user wearing at least a first top item of clothing and a first bottom item of clothing;

defining a first region of the first image data, the first region of the first image data substantially corresponding to pixel data that represents a torso of the user and pixel data that represents legs of the user;

defining a second region of the first image data, the second region of the first image data substantially corresponding to at least a portion of the pixel data that represents the torso of the user;

defining a third region of the first image data, the third region of the first image data substantially corresponding to at least a portion of the pixel data that represents the legs of the user;

storing second image data that depicts the user wearing a second top item of clothing and a second bottom item of clothing;

defining a first region of the second image data, the first region of the second image data substantially corresponding to pixel data that represents the torso of the user and pixel data that represents the legs of the user;

defining a second region of the second image data, the second region of the second image substantially corresponding to at least a portion of the pixel data that represents the torso of the user;

defining a third region of the second image data, the third region of the second image data substantially corresponding to at least a portion of the pixel data that represents the legs of the user;

receiving a request to identify at least one image that depicts an object represented in third image data, the object comprising the first top item of clothing;

analyzing features of the object represented in the third image data with respect to;

(i) features of the first region of the first image data;

(ii) features of the second region of the first image data, (iii) features of the third region of the first image data;

(iv) features of the first region of the second image data;

(v) features of the second region of the second image data, and (vi) features of the third region of the second image data;

determining the first region of the first image data depicts the object; and

outputting an indication that the first image data depicts the object depicted in the third image.

2. The method as recited in claim 1, wherein:

the first image data comprises background pixels and non-background pixels; and

defining the first region of the first image data comprises;

identifying a smallest rectangular area of the first image data that does not exclude a pixel of the non-background pixels; and

designating, as the first region of the first image data, the smallest rectangular area less a top portion of the smallest rectangular area and a bottom portion of the smallest rectangular area.

3. The method as recited in claim 1, wherein:

the first image data comprises background pixels and non-background pixels; and

defining the second region of the first image data comprises;

approximating a waist line of the user in the first image data;

identifying a left outermost vertical line, between the waist line and a top boundary of the first region of the first image data, that includes a non-background pixel;

identifying a right outermost vertical line, between the waist line and the top boundary of the first region of the first image data, that includes a non-background pixel; and

defining, as the second region of the first image data, a rectangular area of the first image data having a top boundary corresponding to the top boundary of the first region, a bottom boundary corresponding to the waist line, a left boundary corresponding to the left outermost line, and a right boundary corresponding to the right outermost line.

4. The method as recited in claim 1, wherein:

the first image data comprises background pixels and non-background pixels; and

defining the third region of the first image data comprises;

approximating a waist line of the user in the first image;

identifying a left outermost vertical line, between the waist line and a bottom boundary of the first region of the first image data, that includes a non-background pixel;

identifying a right outermost vertical line, between the waist line and the bottom boundary of the first region of the first image data, that includes a non-background pixel; and

defining, as the third region of the first image data, a rectangular area of the first image having a top boundary corresponding to the waist line, a bottom boundary corresponding to the bottom boundary of the first region, a left boundary corresponding to the left outermost line, and a right boundary corresponding to the right outermost line.

5. The method as recited in claim 1, further comprising:

defining a fourth region of the first image data, the fourth region of the first image data substantially corresponding to pixel data that represents a center portion of the torso of the user;

defining a fifth region of the first image data, the fifth region of the first image data substantially corresponding to pixel data that represents outside portions of the torso excluding the center portion of the torso;

defining a fourth region of the second image data, the fourth region of the second image data substantially corresponding to at least a portion of the pixel data that represents the center portion of the torso of the user; and

defining a fifth region of the second image data, the fifth region of the second image data substantially corresponding to at least a portion of the pixel data that represents the outside portions of the torso excluding the pixel data that represents the center portion of the torso;

and wherein the analyzing further comprises analyzing the object represented in the third image data with respect to;

(i) the fourth region of the first image data;

(ii) the fifth region of the first image data;

(iii) the fourth region of the second image data; and

(iv) the fifth region of the second image data.

6. A method comprising:

receiving first image data from a client device, the first image data depicting a particular item;

receiving a request to identify additional image data that depicts the particular item;

analyzing pixel data from the first image data corresponding to the particular item relative to pixel data corresponding to a first region of second image data using a pre-trained classifier, the first region substantially corresponding to pixel data that represents a torso of a user depicted in the second image data;

analyzing the pixel data from the first image data corresponding to the particular item relative to pixel data corresponding to a second region of the second image data using the pre-trained classifier, the second region substantially corresponding to pixel data that represents legs of the user depicted in the second image data;

determining the first region depicts the particular item; and

outputting an indication that the second image data depicts the particular item.

7. The method as recited in claim 6, wherein

the pre-trained classifier comprises a pre-trained convolutional neural network.

8. The method as recited in claim 6, wherein the second image data comprises background pixels and non-background pixels, and further comprising:

identifying a top boundary of the first region;

approximating a waist line of the user in the second image data;

identifying a left outermost vertical line, between the waist line and the top boundary, that includes a non-background pixel;

identifying a right outermost vertical line, between the waist line and the top boundary, that includes a non-background pixel; and

defining, as the first region of the first image data, a rectangular area corresponding to the top boundary, the waist line, the left outermost line, and the right outermost line.

9. The method as recited in claim 6, wherein the second image data comprises background pixels and non-background pixels, and further comprising:

identifying a bottom boundary of the second region;

approximating a waist line of the user in the second image data;

identifying a left outermost vertical line, between the waist line and the bottom boundary, that includes a non-background pixel;

identifying a right outermost vertical line, between the waist line and the bottom boundary, that includes a non-background pixel; and

defining, as the second region of the first image data, a rectangular area corresponding to the bottom boundary, the waist line, the left outermost line, and the right outermost line.

10. The method as recited in claim 6, further comprising analyzing the pixel data corresponding to the particular item relative to a third region of the second image data, the third region substantially corresponding to at least a portion of the pixel data that represents the torso of the user and at least a portion of the pixel data that represents the legs of the user.

11. The method as recited in claim 10, wherein the second image data comprises background pixels and non-background pixels, and further comprising:

identifying a smallest rectangular area of the first image data that does not exclude a threshold number of the non-background pixels; and

defining, as the third region of the first image data, the smallest rectangular area less a top portion of the smallest rectangular area and a bottom portion of the smallest rectangular area.

12. The method as recited in claim 6, further comprising analyzing the pixel data corresponding to the particular item relative to a third region of the second image data, the third region substantially corresponding to pixel data that represents a center portion of the torso of the user.

13. The method as recited in claim 6, further comprising analyzing the pixel data corresponding to the particular item relative to a third region of the second image data, the third region substantially corresponding to pixel data that represents outside portions of the torso of the user excluding a center portion of the torso of the user.

14. One or more computing devices comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising;

receiving, from a client device, an indication of a selection of a representation of a particular item presented on a graphical user interface (GUI) of the client device;

receiving a request to identify at least one image that depicts the particular item;

analyzing pixel data corresponding to the particular item relative to pixel data corresponding to a first region of first image data using a pre-trained classifier, the first region substantially corresponding to at pixel data that represents a torso of a user depicted in the first image data;

analyzing the pixel data corresponding to the particular item relative to pixel data corresponding to a second region of the first image data using the pre-trained classifier, the second region substantially corresponding to pixel data that represents legs of the user depicted in the first image data;

determining that at least one of the first region or the second region depicts the particular item; and

outputting an indication that the first image data depicts the particular item.

15. The one or more computing devices as recited in claim 14, wherein

the pre-trained classifier comprises a pre-trained convolutional neural network.

16. The one or more computing devices as recited in claim 14, wherein the first image data comprises background pixels and non-background pixels, and the acts further comprising:

identifying a top boundary of the first region;

approximating a waist line of the user in the first image data;

identifying a left outermost vertical line, between the waist line and the top boundary, that includes a non-background pixel;

identifying a right outermost vertical line, between the waist line and the top boundary, that includes a non-background pixel; and

defining, as the first region of the first image data, a rectangular area corresponding to the top boundary, the waist line, the left outermost line, and the right outermost line.

17. The one or more computing devices as recited in claim 14, wherein the first image data comprises background pixels and non-background pixels, and the acts further comprising:

identifying a bottom boundary of the second region;

approximating a waist line of the user in the first image data;

identifying a left outermost vertical line, between the waist line and the bottom boundary, that includes a non-background pixel;

identifying a right outermost vertical line, between the waist line and the bottom boundary, that includes a non-background pixel; and

defining, as the second region of the first image data, a rectangular area corresponding to the bottom boundary, the waist line, the left outermost line, and the right outermost line.

18. The one or more computing devices as recited in claim 14, the acts further comprising analyzing the pixel data corresponding to the particular item relative to a third region of the first image data, the third region substantially corresponding to at least a portion of the pixel data that represents the torso of the user and at least a portion of the pixel data that represents the legs of the user.

19. The one or more computing devices as recited in claim 18, wherein the first image data comprises background pixels and non-background pixels, and the acts further comprising:

identifying a smallest rectangular area of the first image data that does not exclude a threshold number of the non-background pixels; and

defining, as the third region of the first image data, the smallest rectangular area less a top portion of the smallest rectangular area and a bottom portion of the smallest rectangular area.

20. The one or more computing devices as recited in claim 14, the acts further comprising analyzing the pixel data corresponding to the particular item relative to a third region of the first image data, the third region substantially corresponding to:

(i) a center portion of at least a portion of the pixel data that represents the torso of the user, or (ii) outside portions of at least a portion of the pixel data that represents the torso of the user excluding the center portion of the torso of the user.