Rotation of an image based on image content to correct image orientation

• US 9,280,804 B2
• Filed: 11/16/2012
• Issued: 03/08/2016
• Est. Priority Date: 11/16/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Krishnaswamy, Aravind, Butko, Nicholas
• Primary Examiner(s): Roz, Mark
• Application Number:
  US13/678,850
• Publication Number:
  US 20140140609A1
• Time in Patent Office: 1,208 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 3/608 by skew deformation, e.g. two-pass or three-pass rotation
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

In some implementations, a method rotates images based on image content to correct image orientation. In some implementations, a method includes obtaining one or more identifications of content depicted in an image and determining a current orientation of the content depicted in the image. The current orientation is determined based on the one or more identifications of the content. An amount of rotation for the image is determined that orients the content closer to a predetermined reference orientation than to the current orientation. The image is rotated by the determined amount.

Claims

1. A method comprising:

obtaining an image including image content;

causing a display of the image on a display device;

obtaining one or more identifications of content depicted in the image;

selecting and applying a plurality of techniques to determine a current orientation of the content depicted in the image, wherein the current orientation is determined based on the one or more identifications of the content, wherein the plurality of techniques are selected from a plurality of available techniques;

determining a plurality of different amounts of rotation that orient the content closer to a reference orientation than to the current orientation, wherein the different amounts of rotation are associated with different techniques of the plurality of techniques applied to determine the current orientation, wherein each of the plurality of different amounts of rotation indicates a different rotated orientation of the image;

receiving input from a user in a graphical interface of an image editing program displayed on the display device, the input instructing rotation of the image;

in response to the input, selecting one of the plurality of different amounts of rotation and rotating the image by the selected amount of rotation to the rotated orientation closest to the display of the image among the different rotated orientations in a rotational direction indicated by the user input; and

causing an updated display of the rotated image, including displaying the image at a displayed rotated orientation based on the selected amount of rotation.

2. The method of claim 1 wherein selecting and applying a plurality of techniques includes:

selecting and applying one or more facial orientation techniques to determine an orientation of one or more faces identified in the identified content, including determining a facial orientation axis based on an axis between two eyes of at least one of the one or more faces;

selecting and applying one or more non-facial orientation techniques to determine an orientation of one or more non-facial features identified in the identified content, including determining a non-facial orientation axis based on an edge detection process used on the one or more non-facial features; and

determining a final resultant axis based on combining the facial orientation axis and the non-facial orientation axis, wherein the current orientation of the identified content is determined based on the final resultant axis.

3. The method of claim 1 wherein the content includes multiple different types,

wherein the selecting and applying a plurality of techniques includes selecting and applying a plurality of techniques to determine, for each technique, a particular orientation of the identified content depicted in the image, wherein the selection of the plurality of techniques is based on the types of content depicted in the image, and further comprising:

determining a weight for each of a plurality of the particular orientations, wherein the weights are based on the type of content and include a plurality of different weights; and

combining the weighted current orientations determined by the plurality of techniques to determine the current orientation of the content depicted in the image.

4. The method of claim 1 wherein the reference orientation is aligned with an approximately horizontal axis based on one or more borders or edges of the image.

5. The method of claim 1 wherein determining a current orientation includes using an edge detection technique to find at least one dominant edge in the image, and wherein determining at least one of the different amounts of rotation includes determining an amount of deviation of the at least one dominant edge from the reference orientation.

6. The method of claim 2 further comprising:

determining a different weight for each of the facial orientation axis and the non-facial orientation axis, wherein determining a final resultant axis is based on combining the weighted facial orientation axis and the weighted non-facial orientation axis.

7. The method of claim 3 wherein the type of content associated with the edge detection process includes a depiction of a landscape or an outdoor environment.

8. The method of claim 2 wherein determining the final resultant axis includes determining one of an average axis and a linear progression axis for the facial orientation axis and the non-facial orientation axis.

9. The method of claim 2 wherein selecting and applying one or more non-facial orientation techniques to determine an orientation of the one or more non-facial features identified in the content includes:

selecting at least one edge detection technique from a plurality of available edge detection techniques based on a type of the one or more non-facial features, wherein the at least one selected edge detection technique has a predetermined association with the type of one or more non-facial features.

10. The method of claim 2 wherein the one or more faces are a plurality of faces, and determining a facial orientation axis includes determining an axis between two eyes for each of the plurality of faces depicted in the image, and determining the facial orientation axis as a resultant axis fitted to the determined axes of the plurality of faces.

11. The method of claim 10 wherein determining the resultant axis includes determining one of:

an average axis from the axes of the plurality of faces, a mode axis from the axes of the plurality of faces, and a mean axis from the axes of the plurality of faces.

12. The method of claim 1 wherein causing an updated display of the rotated image includes causing a snapping of the display of the image to the displayed rotated orientation.

13. The method of claim 1 wherein determining a current orientation includes inputting the image to a plurality of classifiers using machine learning to determine the current orientation of the image, wherein each of the classifiers is previously trained to recognize a different rotational position of an image using training images different from the image.

14. The method of claim 1 further comprising:

determining, based on additional user input, whether the updated display of the rotated image is acceptable; and

in response to determining that the updated display of the rotated image is not acceptable;

rotating the image by a new amount of rotation; and

causing a new updated display of the rotated image, including displaying the image at a displayed rotated orientation based on the new amount of rotation.

15. A system comprising:

a storage device; and

at least one processor accessing the storage device and configured to perform operations comprising;

obtaining an image including image content;

causing a display of the image on a display device;

obtaining one or more identifications of content depicted in the image;

selecting and applying a plurality of techniques to determine a current orientation of the content depicted in the image, wherein the current orientation is determined based on the one or more identifications of the content, wherein the plurality of techniques are selected from a plurality of available techniques;

determining a plurality of different amounts of rotation that orient the content closer to a reference orientation than to the current orientation, wherein the different amounts of rotation are associated with different techniques of the plurality of techniques applied to determine the current orientation, wherein each of the plurality of different amounts of rotation indicates a different rotated orientation of the image;

receiving input from a user in a graphical interface of an image editing program displayed on the display device, the input instructing rotation of the image;

in response to the input, selecting one of the plurality of different amounts of rotation and rotating the image by the selected amount of rotation to the rotated orientation closest to the display of the image among the different rotated orientations in a rotational direction indicated by the user input; and

causing an updated display of the rotated image, including displaying the image at a displayed rotated orientation based on the selected amount of rotation.

16. The system of claim 15 wherein the operation of selecting and applying a plurality of techniques includes:

selecting and applying one or more facial orientation techniques to determine an orientation of one or more faces identified in the identified content, including determining a facial orientation axis based on an axis between two eyes of at least one of the one or more faces;

selecting and applying one or more non-facial orientation techniques to determine an orientation of one or more non-facial features identified in the identified content, including determining a non-facial orientation axis based on an edge detection process used on the one or more non-facial features; and

determining a final resultant axis based on combining the facial orientation axis and the non-facial orientation axis, wherein the current orientation of the identified content is determined based on the final resultant axis.

17. The system of claim 15 wherein the content includes multiple different types,

wherein the operation of selecting and applying a plurality of techniques includes selecting and applying a plurality of techniques to determine, for each technique, a particular orientation of the identified content depicted in the image, wherein the selection of the plurality of techniques is based on the types of content depicted in the image, and further comprising:

determining a weight for each of a plurality of the particular orientations, wherein the weights are based on the type of content and include a plurality of different weights; and

combining the weighted current orientations determined by the plurality of techniques to determine the current orientation of the content depicted in the image.

18. A non-transitory computer readable medium having stored thereon software instructions, when executed by a processor, to cause the processor to perform operations including:

obtaining an image including image content;

causing a display of the image on a display device;

obtaining one or more identifications of content depicted in the image;

selecting and applying a plurality of techniques to determine a current orientation of the content depicted in the image, wherein the current orientation is determined based on the one or more identifications of the content, wherein the plurality of techniques are selected from a plurality of available techniques;

determining a plurality of different amounts of rotation that orient the content closer to a reference orientation than to the current orientation, wherein the different amounts of rotation are associated with different techniques of the plurality of techniques applied to determine the current orientation, wherein each of the plurality of different amounts of rotation indicates a different rotated orientation of the image;

receiving input from a user in a graphical interface of an image editing program displayed on the display device, the input instructing rotation of the image;

in response to the input, selecting one of the plurality of different amounts of rotation and rotating the image by the selected amount of rotation to the rotated orientation closest to the display of the image among the different rotated orientations in a rotational direction indicated by the user input; and

causing an updated display of the rotated image, including displaying the image at a displayed rotated orientation based on the selected amount of rotation.

19. The computer readable medium of claim 18 wherein the operation of selecting and applying a plurality of techniques includes:

selecting and applying one or more facial orientation techniques to determine an orientation of one or more faces identified in the identified content, including determining a facial orientation axis based on an axis between two eyes of at least one of the one or more faces;

selecting and applying one or more non-facial orientation techniques to determine an orientation of one or more non-facial features identified in the identified content, including determining a non-facial orientation axis based on an edge detection process used on the one or more non-facial features; and

determining a final resultant axis based on combining the facial orientation axis and the non-facial orientation axis, wherein the current orientation of the identified content is determined based on the final resultant axis.

20. The computer readable medium of claim 18 wherein the content includes multiple different types,

wherein the operation of selecting and applying a plurality of techniques includes selecting and applying a plurality of techniques to determine, for each technique, a particular orientation of the identified content depicted in the image, wherein the selection of the plurality of techniques is based on the types of content depicted in the image, and further comprising:

determining a weight for each of a plurality of the particular orientations, wherein the weights are based on the type of content and include a plurality of different weights; and

combining the weighted current orientations determined by the plurality of techniques to determine the current orientation of the content depicted in the image.