Method and system for automatically cropping images

• US 8,577,182 B1
• Filed: 12/19/2012
• Issued: 11/05/2013
• Est. Priority Date: 07/13/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Goldman, Sally A.
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US13/719,770
• Time in Patent Office: 321 Days
• Field of Search: 382/118, 382/276, 382/293, 382/298, 348/240.99
• US Class Current: 382/298
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

The present invention relates to methods and systems for automatic image cropping. An automatic image cropping system includes a downsampling system, a saliency analyzer, a spatial scatter, and a face detector configured to detecting one or more faces within the image. The saliency value of each pixel not associated with a detected face within the image is adjusted if one or more faces within the image are detected. A search system defines a saliency region based on a total value of saliency values within the saliency region and on a parameter, where the parameter value is based on a relative importance of a false positive pixel to a false negative pixel. The search system also defines a crop region using an aspect ratio that includes the saliency region and determines a position of the crop region based the saliency values within the crop region by performing a one dimensional search.

Claims

1. A client based method for image cropping, comprising:

capturing an image;

uploading the captured image to an image cropping system, wherein the cropping system performs an analysis of the captured image comprising;

downsampling the image to a threshold number of pixels;

computing a saliency value for each pixel based on one or more detected color clusters and a spatial scatter of the one or more color clusters;

detecting one or more faces within the image;

adjusting the saliency value of each pixel not associated with the one or more detected faces within the image if one or more faces within the image are detected; and

defining a saliency region based on a total value of the saliency values within the saliency region and on a parameter value, wherein the parameter value is based on a relative importance of a false positive pixel to a false negative pixel;

receiving cropping coordinates from the image cropping system; and

cropping the captured image based on the received cropping coordinates.

2. The method of claim 1, further comprising causing the cropped image to be displayed.

3. The method of claim 1, wherein the analysis of the captured image comprising further comprises:

determining a position of the crop region, based on the saliency values within the crop region, by performing a one dimensional search; and

cropping the image based on the one dimensional search.

4. The method of claim 3, further comprising comparing a quality value of the cropped image based on the position determined by the one dimensional search to a quality value of the cropped image when positioned at a default position wherein if the quality value of the cropped image at the default position is greater than a threshold amount above the quality value of the cropped image based on the position determined by the one dimensional search then the cropped image is placed at the default position.

5. The method of claim 3, wherein the determining a position of the crop region is based on maximizing the total value of saliency values within the crop region.

6. The method of claim 1, wherein the saliency value of each pixel of the one or more detected faces is greater than other saliency values.

7. The method of claim 6, wherein a total saliency value of a first detected face is greater than a total saliency value of a second detected face.

8. The method of claim 1, further comprising determining the number of color clusters based on a non-parametric density estimation and a mean shift algorithm.

9. The method of claim 1, further comprising producing a confidence measure of the cropped image indicating a likelihood that a portion of a detected face has been cropped.

10. The method of claim 1, further comprising determining a no-crop region, wherein a pixel within the no-crop region is not present within the crop region.

11. The method of claim 10, further comprising expanding the no-crop region to include a border generated around the no-crop region.

12. A client based system for image cropping, comprising:

an image capture device configured to capture an image;

a communication system configured to;

upload the captured image to a server system comprising;

a downsampling system configured to downsample an image to a threshold number of pixels;

a saliency analyzer configured to compute a saliency value for each pixel based on one or more detected color clusters and a spatial scatter of the one or more color clusters;

a face detector configured to detecting one or more faces within the image, wherein the saliency value of each pixel not associated with the one or more detected faces within the image is adjusted if one or more faces within the image is detected; and

a search system configured to;

define a saliency region based on a total value of saliency values within the saliency region and on a parameter, wherein the parameter value is based on a relative importance of a false positive pixel to a false negative pixel; and

define a crop region, based on a specified aspect ratio, which includes the saliency region; and

receive cropping coordinates from the server system; and

a cropping system configured to crop the captured image based on the received cropping coordinates.

13. The system of claim 12, further comprising a presentation device configured to display the cropped image.

14. The system of claim 12, wherein the server system further comprises:

the search system configured to determine a position of the crop region, based on the saliency values within the crop region, by performing a one dimensional search; and

a cropping system configured to crop the image based on the one dimensional search.

15. The system of claim 14, wherein the server system compares a quality value of the cropped image based on the position determined by the one dimensional search to a quality value of the cropped image when positioned at a default position wherein if the quality value of the cropped image at the default position is greater than a threshold amount above the quality value of the cropped image based on the position determined by the one dimensional search then the cropped image is placed at the default position.

16. The system of claim 12, wherein the face detector generates a border around each of the one or more detected faces.

17. The system of claim 12, wherein the search system determines a position of the crop region based on maximizing the total value of saliency values within the crop region.

18. The system of claim 12, wherein the saliency value of each pixel of the one or more detected faces is greater than other saliency values.

19. The system of claim 12, further comprising determining the number of color clusters based on a non-parametric density estimation and a mean shift algorithm.

20. The system of claim 12, wherein the search system is further configured to determine a no-crop region, wherein a pixel within the no-crop region is not present within the crop region.