Region selection for image match

• US 9,990,557 B2
• Filed: 09/21/2017
• Issued: 06/05/2018
• Est. Priority Date: 03/19/2015
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Du, Ming, Dhua, Arnab Sanat Kumar, Cutter, Michael Patrick
• Primary Examiner(s): Seth, Manav
• Assistant Examiner(s): Perlman, David
• Application Number:
  US15/711,580
• Publication Number:
  US 20180012102A1
• Time in Patent Office: 257 Days
• Field of Search: None
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The accuracy of an image matching process can be improved by determining relevant swatch regions of the images, where those regions contain representative patterns of the items of interest represented in those images. Various processes examine a set of visual cues to determine at least one candidate object region, and then collate these regions to determine one or more representative swatch images. For apparel items, this can include locating regions such as an upper body region, torso region, clothing region, foreground region, and the like. Processes such as regression analysis or probability mapping can be used on the collated region data (along with confidence and/or probability values) to determine the appropriate swatch regions.

Claims

1. A system, comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the system to;

receive an image containing a representation of an object;

determine a set of candidate swatch regions based at least in part on a selection of an area of the image;

for each swatch region in the set of candidate swatch regions, identify a subset of pixels of the image based at least in part on the subset of pixels having at least a minimum probability of similarity to the representation of the object in the image;

selecting at least one swatch region from the set of candidate swatch regions based at least in part on the minimum probability of similarity to the representation of the object;

determine at least two or more sub-regions within the at least one swatch region;

collating data for the at least one swatch region to determine a sub-region representative of at least one visual aspect of the object; and

generating a swatch image using pixel values from within the sub-region, the swatch image used to perform a similarity matching process against a set of candidate images.

2. The system of claim 1, wherein the at least two or more sub-regions within the at least one swatch region are determined based at least in part on locations of the two or more sub-regions within the at least one swatch region.

3. The system of claim 1, wherein the instructions when executed further cause the system to:

scan in, from a set of at least one of edges or corners of the image, until one or more edges or contours of the representation of the object are located; and

determine, for each swatch region in the set of candidate swatch regions, whether it is a foreground region or a background region based on the one or more edges or contours of the representation of the object.

4. The system of claim 1, wherein the instructions when executed further cause the system to:

generate a quantized color image having a lower color depth than the image;

analyze the quantized color image to locate pixels having values of skin tones; and

generate a binary skin mask indicating the pixels having the skin tones in order to determine one or more skin regions from the at least two sub-regions.

5. A computer-implemented method, comprising:

receiving an image containing a representation of an object;

determining a set of candidate swatch regions based at least in part on the image, each candidate swatch region of the set corresponding to a subset of pixels of the image determined to have at least a minimum threshold of correspondence to the representation of the object in the image;

collating data for the set of candidate swatch regions to determine at least two or more sub-regions for each subset of pixels;

determining a swatch region representative of at least one visual aspect of the object based at least in part on the at least two or more sub-regions; and

generating a swatch image using pixel values from within the swatch region.

6. The computer-implemented method of claim 5, further comprising:

analyzing the image using a set of modules, each module corresponding to a respective visual cue.

7. The computer-implemented method of claim 6, wherein the respective visual cue for a module of the set of modules is selected from the group consisting of head location, upper body location, skin region location, body pose, hair location, accessory location, limb region, foreground location, and torso location.

8. The computer-implemented method of claim 5, further comprising:

performing a regression analysis to obtain a transfer function that uses the set of candidate swatch regions to determine the swatch region that is optimal for the object represented in the image.

9. The computer-implemented method of claim 8, further comprising:

utilizing at least one of a linear regression model or a random forest algorithm in the performing of the regression analysis.

10. The computer-implemented method of claim 8, further comprising:

determining at least one confidence value for each of the set of candidate swatch regions for use in the performing of the regression analysis, at least one confidence value determined based at least in part upon a number of pixels within a respective candidate swatch region having a value corresponding to a background color or a skin tone.

11. The computer-implemented method of claim 5, further comprising:

generating a probability map based on probability data for the set of candidate swatch regions, the object region being selected from a portion of the probability map having at least a minimum probability value.

12. The computer-implemented method of claim 11, wherein the probability data for a selected candidate swatch region of the set comprises a single probability value or a distribution of probability values.

13. The computer-implemented method of claim 5, further comprising:

determining, from within the object region, a set of candidate swatch regions; and

comparing the candidate swatch regions, within the set of candidate swatch regions, to remove a swatch region of each pair of redundant swatch regions, in order to determine the swatch region, the swatch region capable of being one of a set of swatch regions generated for the image.

14. The computer-implemented method of claim 5, further comprising:

performing an image match process against a set of object images using the swatch image; and

providing results of the image match process to a user having provided the image.

15. A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor of a computing system, causes the computing system to:

receive an image containing a representation of an object;

determine a set of candidate swatch regions based at least in part on the image, each candidate swatch region of the set of candidate swatch regions corresponding to a subset of pixels of the image determined to have at least a minimum threshold of correspondence to the representation of the object in the image;

collate data for the set of candidate swatch regions to determine at least two or more sub-regions for each subset of pixels;

determine a swatch region representative of at least one visual aspect of the object based at least in part on the at least two or more sub-regions; and

generate a swatch image using pixel values from within the swatch region.

16. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

analyze the image based on a set of visual cues, the set of visual cues including at least one of head location, upper body location, skin region location, body pose, hair location, foreground location, and torso location.

17. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

perform a regression analysis on collated data for the set of candidate swatch regions to determine the object region.

18. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

generate a probability map based on probability data for the set of candidate swatch regions, the object region being selected from a portion of the probability map having at least a minimum probability value.

19. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

determine, from within the object region, a set of candidate swatch regions; and

compare the candidate swatch regions within the set of candidate swatch regions to remove a swatch region of each pair of redundant swatch regions to determine the swatch region, the swatch region capable of being one of a set of swatch regions generated for the image.

20. The non-transitory computer readable storage medium of claim 15, wherein the instructions, when executed by the at least one processor, further causes the computing system to:

determine at least one confidence value for each of the set of candidate swatch regions for use in the performing of the regression analysis, at least one confidence value determined based at least in part upon a number of pixels within a respective candidate swatch region having a value corresponding to a background color or a skin tone.