VISUAL CLOTHING RETRIEVAL

• US 20140314313A1
• Filed: 04/17/2013
• Published: 10/23/2014
• Est. Priority Date: 04/17/2013
• Status: Active Grant

• Current Assignee: Yahoo Assets LLC
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Kennedy, Lyndon, Li, Jia, Kalantidis, Ioannis
• Granted Patent: US 9,460,518 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Techniques are provided for efficiently identifying relevant product images based on product items detected in a query image. In general, a query image may represent a digital image in any format that depicts a human body and one or more product items. For example, a query image may be an image for display on a webpage, an image captured by a user using a camera device, or an image that is part of a media content item, such as a frame from a video. Product items may be detected in a query image by segmenting the query image into a plurality of image segments and clustering one or more of the plurality image segments into one or more image segment clusters. The resulting image segments and image segment clusters may be used to search for visually similar product images.

Claims

1. A computer-implemented method comprising:

segmenting an image into a plurality of image segments;

clustering one or more image segments from the plurality of image segments into one or more image segment clusters, wherein at least one of the one or more image segment clusters comprises two or more visually similar image segments;

for a particular image segment cluster of the one or more image segment clusters;

determining, for the particular image segment cluster, a particular product class of a plurality of product classes;

retrieving, from an image database, one or more product images that are associated with the particular product class and that are selected based on a visual similarity between the one or more product images and the particular image segment cluster;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, further comprising:

generating, for the particular image segment cluster, a representation of a spatial location of the particular image segment cluster;

determining the particular product class for the particular image segment cluster based on a training set of images comprising one or more mappings between particular spatial locations and particular product classes.

3. The method of claim 2, wherein the spatial location is defined relative to a location on a human body.

4. The method of claim 1, wherein the two or more visually similar image segments are determined to be visually similar based on characteristics of the two or more visually similar image segments including image color and image texture.

5. The method of claim 1, wherein the two or more visually similar image segments are spatially separated in the image.

6. The method of claim 1, further comprising, for each of the one or more product images:

segmenting the product image into a product image segment and a non-product image segment;

storing the product image segment in an image database in association with a product class.

7. The method of claim 1, further comprising:

generating an articulated pose estimation for the image, the articulated pose estimation comprising one or more image regions corresponding to a location on a human body;

generating, based on the articulated pose estimation, a product probability map comprising one or more values indicating a probability that particular regions of the image depict a product item.

8. The method of claim 7, further comprising generating, based on the product probability map, an image mask configured to mask particular regions of the image that are not likely to depict a product item.

9. The method of claim 1, wherein a product class of the plurality of product classes represents a type of clothing item.

10. The method of claim 1, wherein the image comprises a depiction of a human and one or more clothing items.

11. A non-transitory computer-readable medium carrying instructions which, when processed by one or more processors, cause:

segmenting an image into a plurality of image segments;

clustering one or more image segments from the plurality of image segments into one or more image segment clusters, wherein at least one of the one or more image segment clusters comprises two or more visually similar image segments;

for a particular image segment cluster of the one or more image segment clusters;

determining, for the particular image segment cluster, a particular product class of a plurality of product classes;

retrieving, from an image database, one or more product images that are associated with the particular product class and that are selected based on a visual similarity between the one or more product images and the particular image segment cluster.

12. The non-transitory computer-readable medium as recited in claim 11, further comprising additional instructions which, when processed by the one or more processors, cause:

generating, for the particular image segment cluster, a representation of a spatial location of the particular image segment cluster;

determining the particular product class for the particular image segment cluster based on a training set of images comprising one or more mappings between particular spatial locations and particular product classes.

13. The non-transitory computer-readable medium as recited in claim 12, wherein the spatial location is defined relative to a location on a human body.

14. The non-transitory computer-readable medium as recited in claim 11, wherein the two or more visually similar image segments are determined to be visually similar based on characteristics of the two or more visually similar image segments including image color and image texture.

15. The non-transitory computer-readable medium as recited in claim 11, wherein the two or more visually similar image segments are spatially separated in the image.

16. The non-transitory computer-readable medium as recited in claim 11, further comprising additional instructions which, when processed by the one or more processors, cause, for each of the one or more product images:

segmenting the product image into a product image segment and a non-product image segment;

storing the product image segment in an image database in association with a product class.

17. The non-transitory computer-readable medium as recited in claim 11, further comprising additional instructions which, when processed by the one or more processors, cause:

generating an articulated pose estimation for the image, the articulated pose estimation comprising one or more image regions corresponding to a location on a human body;

generating, based on the articulated pose estimation, a product probability map comprising one or more values indicating a probability that particular regions of the image depict a product item.

18. The non-transitory computer-readable medium as recited in claim 17, further comprising additional instructions which, when processed by the one or more processors, cause generating, based on the product probability map, an image mask configured to mask particular regions of the image that are not likely to depict a product item.

19. The non-transitory computer-readable medium as recited in claim 11, wherein a product class of the plurality of product classes represents a type of clothing item.

20. The non-transitory computer-readable medium as recited in claim 11, wherein the image comprises a depiction of a human and one or more clothing items.