Method and apparatus for characterizing an image

• US 8,873,838 B2
• Filed: 03/14/2013
• Issued: 10/28/2014
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Suleyman, Mustafa, Barwinski, Marek, Nair, Arun, Rusu, Andrei-Alexandru, Carlton Chu, Chia-Yueh, Coppin, Benjamin Kenneth
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): MOYER, ANDREW M
• Application Number:
  US13/804,257
• Publication Number:
  US 20140270488A1
• Time in Patent Office: 593 Days
• US Class Current: 382/157
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

The present invention relates to a method and system for characterizing an image. The characterization may then be used to conduct a search for similar images, for example using a learning system trained using previously characterized images. A face may be identified within the image and a subsection extracted from said image which does not contain said face. At least one fixed size patch is taken from said extracted subsection; and input into said learning network to characterize said image.

Claims

1. A computer-implemented method comprising:

maintaining characterization data for each of a plurality of images, the characterization data for each image characterizing clothing worn by a person pictured in the image, the characterization data for each image including a feature record that includes a respective number for each of a plurality of pattern categories, the respective number for each of the plurality of pattern categories representing an extent to which the pattern category is contained in the clothing worn by the person pictured in the image;

receiving a request from a user, the request identifying an input image;

generating characterization data for the input image, the characterization data for the input image characterizing clothing worn by a person pictured in the input image, wherein generating the characterization data for the input image comprises;

identifying a face within the input image;

extracting a clothing subsection from the input image, wherein extracting the clothing subsection from the input image comprises removing a background from the input image, removing the face from the input image, and removing other skin areas from the input image;

determining a size of the face in the input image;

rescaling the clothing subsection based on the size of the face in the input image;

taking one or more fixed size patches from the rescaled clothing subsection;

gray scaling each of the fixed size patches to remove color from the fixed size patch; and

processing only the gray scaled fixed size patches using a learning network to generate a feature record for the input image, the feature record for the input image including a respective number for each of a plurality of pattern categories, the number for each of the plurality of pattern categories representing an extent to which the corresponding pattern category is contained in the clothing worn by the person pictured in the input image;

selecting, using the characterization data for the input image and the maintained characterization data, one or more similar images to the input image from the plurality of images; and

providing the similar images for presentation to the user in response to the request.

2. The method of claim 1, wherein extracting the clothing subsection from the input image comprises analyzing the identified face to determine a skin tone for the face.

3. The method of claim 2, wherein removing the other skin areas from the input image comprises removing the other skin areas using the skin tone.

4. The method of claim 3, wherein extracting the clothing subsection from the input image comprises applying a median filter to replace any speckles of skin tone.

5. The method of claim 2, wherein extracting the clothing subsection from the input image comprises using a skin tone mask to remove any patches from the input image having a similar color to the skin tone.

6. The method of claim 5, wherein removing the background comprises applying a background mask and wherein the skin tone mask is combined with the background mask.

7. The method of claim 1, wherein each patch is 32×

32 pixels.

8. The method of claim 1, comprising extracting a plurality of fixed size patches from within the extracted subsection and processing each of the plurality of patches using the learning system.

9. The method of claim 1, wherein the learning system is a neural network.

10. The method of claim 9, wherein the neural network comprises at least one set of convolutional neural network filters.

11. The method of claim 10, wherein the neural network comprises a first set of convolutional filters which generates a first set of outputs and a second set of convolutional filters which generate a second set of outputs derived from the first set of outputs.

12. The method of claim 11, wherein the first set of outputs are input to the second set of convolutional filters via a non-linear layer.

13. The method of claim 11, wherein the first set of outputs are input to the second set of convolutional filters via a downsampling layer.

14. The method of claim 11, wherein each of the second set of convolutional filters receives a plurality of outputs selected from the first set of outputs.

15. The method of claim 11, wherein the second set of outputs are input to another learning network.

16. The method of claim 1, further comprising training the learning system using a set of training images, wherein, for each training image, the pattern categories contained in the clothing worn by the person pictured in the training image are known.

17. The method of claim 1, wherein the characterization data for each image further includes a color histogram for the image that characterizes colors of the clothing worn by the person pictured in the image.

18. The method of claim 17, wherein generating the characterization data for the input image further comprises processing the clothing subsection using a color analysis module to generate a color histogram for the clothing worn by the person pictured in the input image, and wherein selecting one or more similar images to the input image from the plurality of images comprises selecting images that have a similar feature record and a similar color histogram to the feature record and the color histogram for the input image as the similar images to the input image.

19. The method of claim 1, wherein the characterization data for each image includes a face to clothing distance for the image, the face to clothing distance being a distance in the image between a face of the person pictured in the image and a clothing subsection of the image.

20. The method of claim 19, wherein generating the characterization data for the input image further comprises determining a distance in the input image between the face identified in the input image and the clothing subsection extracted from the input image, and wherein selecting one or more similar images to the input image from the plurality of images comprises selecting images that have a similar face to clothing distance to the distance in the input image between the face identified in the input image and the clothing subsection extracted from the input image.

21. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

maintaining characterization data for each of a plurality of images, the characterization data for each image characterizing clothing worn by a person pictured in the image, the characterization data for each image including a feature record that includes a respective number for each of a plurality of pattern categories, the respective number for each of the plurality of pattern categories representing an extent to which the pattern category is contained in the clothing worn by the person pictured in the image;

receiving a request from a user, the request identifying an input image;

generating characterization data for the input image, the characterization data for the input image characterizing clothing worn by a person pictured in the input image, wherein generating the characterization data for the input image comprises;

identifying a face within the input image;

extracting a clothing subsection from the input image, wherein extracting the clothing subsection from the input image comprises removing a background from the input image, removing the face from the input image, and removing other skin areas from the input image;

determining a size of the face in the input image;

rescaling the clothing subsection based on the size of the face in the input image;

taking at least one fixed size patch from the rescaled clothing subsection;

gray scaling each of the fixed size patches to remove color from the fixed size patch; and

processing only the gray scaled fixed size patches using a learning network to generate a feature record for the input image, the feature record for the input image including a respective number for each of a plurality of pattern categories, the number for each of the plurality of pattern categories representing an extent to which the corresponding pattern category is contained in the clothing worn by the person pictured in the input image;

selecting, using the characterization data for the input image and the maintained characterization data, one or more similar images to the input image from the plurality of images; and

providing the similar images for presentation to the user in response to the request.

22. A computer-readable medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

maintaining characterization data for each of a plurality of images, the characterization data for each image characterizing clothing worn by a person pictured in the image, the characterization data for each image including a feature record that includes a respective number for each of a plurality of pattern categories, the respective number for each of the plurality of pattern categories representing an extent to which the pattern category is contained in the clothing worn by the person pictured in the image;

receiving a request from a user, the request identifying an input image;

generating characterization data for the input image, the characterization data for the input image characterizing clothing worn by a person pictured in the input image, wherein generating the characterization data for the input image comprises;

identifying a face within the input image;

extracting a clothing subsection from the input image, wherein extracting the clothing subsection from the input image comprises removing a background from the input image, removing the face from the input image, and removing other skin areas from the input image;

determining a size of the face in the input image;

rescaling the clothing subsection based on the size of the face in the input image;

taking at least one fixed size patch from the rescaled clothing subsection;

gray scaling each of the fixed size patches to remove color from the fixed size patch; and

processing only the gray scaled fixed size patches using a learning network to generate a feature record for the input image, the feature record for the input image including a respective number for each of a plurality of pattern categories, the number for each of the plurality of pattern categories representing an extent to which the corresponding pattern category is contained in the clothing worn by the person pictured in the input image;

selecting, using the characterization data for the input image and the maintained characterization data, one or more similar images to the input image from the plurality of images; and

providing the similar images for presentation to the user in response to the request.