Visual clothing retrieval

• US 9,460,518 B2
• Filed: 04/17/2013
• Issued: 10/04/2016
• Est. Priority Date: 04/17/2013
• Status: Active Grant

• Current Assignee: Yahoo Assets LLC
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Kennedy, Lyndon, Li, Jia, Kalantidis, Ioannis
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): Lesnick, Alexander J
• Application Number:
  US13/865,142
• Publication Number:
  US 20140314313A1
• Time in Patent Office: 1,266 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Techniques are provided for efficiently identifying relevant product images based on product items detected in a query image. In general, a query image may represent a digital image in any format that depicts a human body and one or more product items. For example, a query image may be an image for display on a webpage, an image captured by a user using a camera device, or an image that is part of a media content item, such as a frame from a video. Product items may be detected in a query image by segmenting the query image into a plurality of image segments and clustering one or more of the plurality image segments into one or more image segment clusters. The resulting image segments and image segment clusters may be used to search for visually similar product images.

Claims

1. A computer-implemented method comprising:

without segmenting an image generating a pose estimation, determining one or more regions of the image that are not likely to depict a product item;

generating a masked image that masks the one or more regions;

segmenting the masked image into a plurality of image segments based on shared characteristics of pixels that comprise the masked image, each image segment comprising a plurality of pixels that;

form a continuous region of the masked image, and

share a characteristics,

wherein two or more image segments of the plurality of image segments depict a same product;

clustering the two or more image segments from the plurality of image segments into an image segment cluster;

determining, for the image segment cluster, a particular product class of a plurality of product classes;

retrieving, from an image database, one or more product images that are associated with the particular product class and that are selected based on a visual similarity between the one or more product images and the image segment cluster;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, further comprising:

generating, for the image segment cluster, a representation of a spatial location of the image segment cluster;

determining the particular product class for the image segment cluster based on a training set of images comprising one or more mappings between particular spatial locations and particular product classes.

3. The method of claim 2, wherein the spatial location is defined relative to a location on a human body.

4. The method of claim 1, wherein the two or more image segments are determined to be visually similar based on characteristics of the two or more image segments including image color and image texture.

5. The method of claim 1, wherein the two or more image segments are spatially separated in the image.

6. The method of claim 1, further comprising, for each of the one or more product images:

segmenting the product image into a product image segment and a non-product image segment;

storing the product image segment in an image database in association with a product class.

7. The method of claim 1, wherein a product class of the plurality of product classes represents a type of clothing item.

8. The method of claim 1, wherein the image comprises a depiction of a human and one or more clothing items.

9. A non-transitory computer-readable medium carrying instructions which, when processed by one or more processors, cause:

without segmenting an image generating a pose estimation, determining one or more regions of the image that are not likely to depict a product item;

generating a masked image that masks the one or more regions;

segmenting the masked image into a plurality of image segments based on shared characteristics of pixels that comprise the masked image, each image segment comprising a plurality of pixels that;

form a continuous region of the masked image, and

share a characteristics,

wherein two or more image segments of the plurality of image segments depict a same product;

clustering the two or more image segments from the plurality of image segments into an image segment cluster;

determining, for the image segment cluster, a particular product class of a plurality of product classes;

retrieving, from an image database, one or more product images that are associated with the particular product class and that are selected based on a visual similarity between the one or more product images and the image segment cluster.

10. The non-transitory computer-readable medium as recited in claim 9, further comprising additional instructions which, when processed by the one or more processors, cause:

generating, for the image segment cluster, a representation of a spatial location of the image segment cluster;

determining the particular product class for the image segment cluster based on a training set of images comprising one or more mappings between particular spatial locations and particular product classes.

11. The non-transitory computer-readable medium as recited in claim 10, wherein the spatial location is defined relative to a location on a human body.

12. The non-transitory computer-readable medium as recited in claim 9, wherein the two or more image segments are determined to be visually similar based on characteristics of the two or more image segments including image color and image texture.

13. The non-transitory computer-readable medium as recited in claim 9, wherein the two or more image segments are spatially separated in the image.

14. The non-transitory computer-readable medium as recited in claim 9, further comprising additional instructions which, when processed by the one or more processors, cause, for each of the one or more product images:

segmenting the product image into a product image segment and a non-product image segment;

storing the product image segment in an image database in association with a product class.

15. The non-transitory computer-readable medium as recited in claim 9, wherein a product class of the plurality of product classes represents a type of clothing item.

16. The non-transitory computer-readable medium as recited in claim 9, wherein the image comprises a depiction of a human and one or more clothing items.

17. A system comprising:

an image database storing one or more product images;

a computer comprising a processor and a memory, the memory storing one or more sequences of instructions which when executed cause the computer to perform;

without segmenting an image generating a pose estimation, determining one or more regions of the image that are not likely to depict a product item;

generating a masked image that masks the one or more regions;

segmenting the masked image into a plurality of image segments based on shared characteristics of pixels that comprise the masked image, each image segment comprising a plurality of pixels that;

form a continuous region of the masked image, and

share a characteristics,

wherein two or more image segments of the plurality of image segments depict a same product;

clustering the two or more image segments from the plurality of image segments into an image segment cluster;

determining, for the image segment cluster, a particular product class of a plurality of product classes;

retrieving, from an image database, one or more product images that are associated with the particular product class and that are selected based on a visual similarity between the one or more product images and the image segment cluster.

18. The system of claim 17, the one or more sequences of instructions further causing the computer to perform:

generating, for the image segment cluster, a representation of a spatial location of the image segment cluster;

determining the particular product class for the image segment cluster based on a training set of images comprising one or more mappings between particular spatial locations and particular product classes.

19. The system of claim 18, wherein the spatial location is defined relative to a location on a human body.

20. The system of claim 17, the one or more sequences of instructions further causing the computer to perform:

segmenting the product image into a product image segment and a non-product image segment;

storing the product image segment in an image database in association with a product class.