Region selection for image match

• US 9,798,949 B1
• Filed: 03/19/2015
• Issued: 10/24/2017
• Est. Priority Date: 03/19/2015
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Du, Ming, Dhua, Arnab Sanat Kumar, Cutter, Michael Patrick
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Perlman, David
• Application Number:
  US14/662,896
• Time in Patent Office: 950 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The accuracy of an image matching process can be improved by determining relevant swatch regions of the images, where those regions contain representative patterns of the items of interest represented in those images. Various processes examine a set of visual cues to determine at least one candidate object region, and then collate these regions to determine one or more representative swatch images. For apparel items, this can include locating regions such as an upper body region, torso region, clothing region, foreground region, and the like. Processes such as regression analysis or probability mapping can be used on the collated region data (along with confidence and/or probability values) to determine the appropriate swatch regions.

Claims

1. A system, comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

obtain an image including a representation of an object, the object including an apparel item worn by a person, the image further including a region of a background color;

determine a foreground region by locating a boundary around the representation of the object in the image, a portion of the image outside the bounding region having the background color;

select a foreground swatch region from within the foreground region;

locate, using a first object detection algorithm, a representation of a face in the image;

determine a torso swatch region of the image based at least in part upon a location of the representation of the face in the image;

determine, using a second object detection algorithm, an upper body region of the image;

determine one or more skin regions in the image including pixels with color values corresponding to skin tones;

determine an apparel swatch region by locating a subset of the pixels positioned with respect to the one or more skin regions;

select at least one of the foreground swatch region, the torso swatch region, the upper body region, and the apparel swatch region;

collate data for the selection of the at least one of the foreground swatch region, the torso swatch region, the upper body region, and the apparel swatch region to generate a distribution of probabilities corresponding to pixel locations in the image;

determine, based at least in part upon the distribution of probabilities and the selection, a final swatch region corresponding to a subset of contiguous pixels in the image having at least a minimum probability level; and

provide a swatch image, corresponding to the final swatch region, to a matching component capable of using the swatch image to perform a matching process against a set of candidate images.

2. The system of claim 1, wherein the instructions when executed further cause the system to:

determine, from the distribution of probabilities, a set of candidate swatch regions; and

compare the candidate swatch regions, within the set of candidate swatch regions, to remove a swatch region of each pair of redundant swatch regions, in order to determine the swatch region to be selected, the swatch region capable of being one of a set of swatch regions generated for the image.

3. The system of claim 1, wherein the instructions when executed further cause the system to:

scan in, from a set of at least one of edges or corners of the image, until one or more edges or contours of the representation of the object are located, in order to determine the foreground region.

4. The system of claim 1, wherein the instructions when executed further cause the system to:

generate a quantized color image having a lower color depth than the obtained image;

analyze the quantized color image to locate regions of the pixels having values of the skin tones; and

generate a binary skin mask indicating the regions having the skin tones in order to determine the one or more skin regions.

5. A computer-implemented method, comprising:

receiving, by one or more computer systems, an image containing a representation of an object;

determining a selection of an area of the image;

analyzing the image to determine a set of candidate swatch regions based at least in part on the selection of the area of the image, each candidate swatch region of the set corresponding to a subset of pixels of the image determined to have at least a minimum probability of similarity to the representation of the object in the image;

selecting at least one swatch region from the set of candidate swatch regions based at least in part on the minimum probability of similarity to the representation of the object in the image, the set of candidate swatch regions including at least two of a foreground swatch region, a torso swatch region, an upper body swatch region, or an apparel swatch region;

collating data for the selection from the set of candidate swatch regions to determine a swatch region representative of at least one visual aspect of the object; and

generating a swatch image using pixel values from within the swatch region, the swatch image used to perform an similarity matching process against a set of candidate images.

6. The computer-implemented method of claim 5, further comprising:

analyzing the image using a set of modules, each module corresponding to a respective visual cue.

7. The computer-implemented method of claim 6, wherein the respective visual cue for a module of the set of modules is selected from the group consisting of head location, upper body location, skin region location, body pose, hair location, accessory location, limb region, foreground location, and torso location.

8. The computer-implemented method of claim 5, further comprising:

performing a regression analysis to obtain a transfer function that uses the set of candidate swatch regions to determine the swatch region that is optimal for the object represented in the image.

9. The computer-implemented method of claim 8, further comprising:

utilizing at least one of a linear regression model or a random forest algorithm in the performing of the regression analysis.

10. The computer-implemented method of claim 8, further comprising:

determining at least one confidence value for each of the set of candidate swatch regions for use in the performing of the regression analysis, at least one confidence value determined based at least in part upon a number of pixels within a respective candidate swatch region having a value corresponding to a background color or a skin tone.

11. The computer-implemented method of claim 5, further comprising:

generating a probability map based on probability data for the set of candidate swatch regions, the object region being selected from a portion of the probability map having at least a minimum probability value.

12. The computer-implemented method of claim 11, wherein the probability data for a selected candidate swatch region of the set comprises a single probability value or a distribution of probability values.

13. The computer-implemented method of claim 5, further comprising:

determining, from within the object region, a set of candidate swatch regions; and

comparing the candidate swatch regions, within the set of candidate swatch regions, to remove a swatch region of each pair of redundant swatch regions, in order to determine the swatch region, the swatch region capable of being one of a set of swatch regions generated for the image.

14. The computer-implemented method of claim 5, further comprising:

performing an image match process against a set of object images using the swatch image; and

providing results of the image match process to a user having provided the image.

15. A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor of a computing system, causes the computing system to:

receive an image containing a representation of an object;

determine a selection of an area of the image;

analyze the image to determine a set of candidate swatch regions based at least in part on the selection of the area of the image, each candidate swatch region of the set corresponding to a subset of pixels of the image determined to have at least a minimum probability of similarity to the representation of the object in the image;

select at least one swatch region from the set of candidate swatch regions based at least in part on the minimum probability of similarity to the representation of the object in the image, the set of candidate swatch regions including at least two of a foreground swatch region, a torso swatch region, an upper body swatch region, or an apparel swatch region;

collate data for the selection from the set of candidate swatch regions to determine an object region in the image;

select, from within the object region, a swatch region representative of at least one visual aspect of the object; and

generate a swatch image using pixel values from within the swatch region, the swatch image used to perform an similarity matching process against a set of candidate images.

16. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

analyze the image based on a set of visual cues, the set of visual cues including at least one of head location, upper body location, skin region location, body pose, hair location, foreground location, and torso location.

17. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

perform a regression analysis on collated data for the set of candidate swatch regions to determine the object region.

18. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

generate a probability map based on probability data for the set of candidate swatch regions, the object region being selected from a portion of the probability map having at least a minimum probability value.

19. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

determine, from within the object region, a set of candidate swatch regions; and

compare the candidate swatch regions within the set of candidate swatch regions to remove a swatch region of each pair of redundant swatch regions to determine the swatch region, the swatch region capable of being one of a set of swatch regions generated for the image.

20. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

determine at least one confidence value for each of the set of candidate swatch regions for use in the performing of a regression analysis, at least one confidence value determined based at least in part upon a number of pixels within a respective candidate swatch region having a value corresponding to a background color or a skin tone.