Automatic correction of skew in natural images and video

• US 9,076,242 B2
• Filed: 03/14/2013
• Issued: 07/07/2015
• Est. Priority Date: 07/19/2012
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Baheti, Pawan Kumar, Barman, Kishor K., Acharya, Hemanth P.
• Primary Examiner(s): MOE, AUNG SOE
• Application Number:
  US13/831,237
• Publication Number:
  US 20140022406A1
• Time in Patent Office: 845 Days
• Field of Search: 348/222.1, 382171-180, 382/289, 382/229
• US Class Current: 1/1
• CPC Class Codes: G06T 3/608 by skew deformation, e.g. two-pass or three-pass rotation
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1478 of characters or characters lines
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image

Abstract

An electronic device and method use a camera to capture an image of an environment outside followed by identification of regions therein. A subset of the regions is selected, based on attributes of the regions, such as aspect ratio, height, and variance in stroke width. Next, a number of angles that are candidates for use as skew of the image are determined (e.g. one angle is selected for each region. based on peakiness of a histogram of the region, evaluated at different angles). Then, an angle that is most common among these candidates is identified as the angle of skew of the image. The just-described identification of skew angle is performed prior to classification of any region as text or non-text. After skew identification, at least all regions in the subset are rotated by negative of the skew angle, to obtain skew-corrected regions for use in optical character recognition.

Claims

1. A method to correct skew in an image of a scene of real world, the method comprising:

automatically selecting a subset of first regions by filtering out second regions, from among a set of regions identified in the image, at least based on one or more attributes related to geometry of the regions;

automatically determining a plurality of candidate angles of skew, at least based on pixels in a particular region in the subset of first regions;

automatically identifying as an angle of skew of the image, a most common candidate angle among the plurality of candidate angles; and

automatically rotating by negative of the angle of skew of the image, at least the subset of first regions;

wherein the automatically selecting, the automatically determining, the automatically identifying and the automatically rotating are performed by at least one processor.

2. The method of claim 1 wherein:

the one or more attributes related to the geometry of the regions includes one or more attributes selected from a group consisting of aspect ratio, height, and variance in width; and

the automatically selecting comprises sorting the set of regions based on at least one of the one or more attributes related to the geometry of the regions.

3. The method of claim 1 wherein:

height is one of the one or more attributes related to the geometry of the regions; and

the automatically selecting comprises comparing against a predetermined threshold H, the height of the particular region.

4. The method of claim 1 wherein:

variance in width is one of the one or more attributes related to the geometry of the regions; and

the automatically selecting comprises comparing the variance in width of a portion of the particular region against a predetermined threshold V.

5. The method of claim 4 wherein:

the width used in comparing is a stroke width of the particular region.

6. The method of claim 1 wherein:

the automatically determining comprises preparing a histogram of counters of pixels in the particular region, to identify for the particular region a peak and repeating the preparing of a histogram at multiple angles relative to the particular region, to detect a largest peak and to identify the angle at which the largest peak is detected as a candidate angle.

7. The method of claim 1 wherein:

the automatically determining comprises checking whether at a candidate angle, one or more attributes of projection profiles meet at least one test for presence of a straight line of pixels in the particular region.

8. The method of claim 1 wherein:

the automatically selecting, the automatically determining, the automatically identifying and the automatically rotating are performed prior to classification of the subset of first regions as text or non-text.

9. At least one non-transitory computer readable storage media comprising a plurality of instructions to be executed by at least one processor to correct skew in an image of a scene of real world, the plurality of instructions comprising:

first instructions to select a subset of first regions by filtering out second regions, from among a set of regions identified in the image, at least based on one or more attributes related to geometry of the regions;

second instructions to determine a plurality of candidate angles of skew, at least based on pixels in a particular region in the subset of first regions;

third instructions to identify as an angle of skew of the image, a most common candidate angle among the plurality of candidate angles; and

fourth instructions to rotate by negative of the angle of skew of the image, at least the subset of first regions.

10. The at least one non-transitory computer readable storage media of claim 9 wherein:

the one or more attributes related to the geometry of the regions includes one or more attributes selected from a group consisting of aspect ratio, height, and variance in stroke width; and

the first instructions comprise instructions to sort the set of regions based on at least one of the one or more attributes related to the geometry of the regions.

11. The at least one non-transitory computer readable storage media of claim 9 wherein:

height is one of the one or more attributes related to the geometry of the regions; and

the first instructions comprise instructions to compare against a predetermined threshold H, the height of the particular region.

12. The at least one non-transitory computer readable storage media of claim 9 wherein:

variance in width is one of the one or more attributes related to the geometry of the regions; and

the first instructions comprise instructions to compare the variance in width of a portion of the particular region against a predetermined threshold V.

13. The at least one non-transitory computer readable storage media of claim 12 wherein:

the width used by the instructions to compare is a stroke width of the particular region.

14. The at least one non-transitory computer readable storage media of claim 9 wherein:

the second instructions comprise instructions to prepare a histogram of counters of pixels in the particular region, to identify for the particular region a peak and repeat execution of the instructions to prepare a histogram at multiple angles relative to the particular region, to detect a largest peak and to identify the angle at which the largest peak is detected as a candidate angle.

15. The at least one non-transitory computer readable storage media of claim 9 wherein:

the second instructions comprise instructions to check whether at a candidate angle, one or more attributes of projection profiles meet at least one test for presence of a straight line of pixels in the particular region.

16. The at least one non-transitory computer readable storage media of claim 9 wherein:

the first instructions, the second instructions, the third instructions, and the fourth instructions are configured to be executed prior to classification of the subset of first regions as text or non-text.

17. A mobile device to decode text in real world images, the mobile device comprising:

a camera;

a memory operatively connected to the camera to receive at least an image therefrom, the image comprising one or more text regions;

at least one processor operatively connected to the memory to execute a plurality of instructions stored in the memory;

wherein the plurality of instructions cause the at least one processor to;

select a subset of first regions by filtering out second regions, from among a set of regions identified in the image, at least based on one or more attributes related to geometry of the regions;

determine a plurality of candidate angles of skew, at least based on of pixels in a particular region in the subset of first regions;

identify as an angle of skew of the image, a most common candidate angle among the plurality of candidate angles; and

rotate by negative of the angle of skew of the image, at least the subset of first regions.

18. The mobile device of claim 17 wherein the one or more attributes related to the geometry of the regions includes one or more attributes selected from a group consisting of aspect ratio, height, and variance in stroke width and wherein the at least one processor is further configured to:

sort the set of regions based on at least one of the one or more attributes related to the geometry of the regions.

19. The mobile device of claim 17 wherein the at least one processor is further configured to:

height is one of the one or more attributes related to the geometry of the regions; and

compare against a predetermined threshold H, the height of the particular region.

20. The mobile device of claim 17 wherein the at least one processor is further configured to:

variance in width is one of the one or more attributes related to the geometry of the regions; and

compare the variance in width of a portion of the particular region against a predetermined threshold V.

21. The mobile device of claim 20 wherein:

the width used by the at least one processor to compare is a stroke width of the particular region.

22. The mobile device of claim 17 wherein:

the plurality of instructions comprise instructions to prepare a histogram of counters of pixels in the particular region, to identify for the particular region a peak and repeat execution of the instructions to prepare a histogram at multiple angles relative to the particular region, to detect a largest peak and to identify the angle at which the largest peak is detected as a candidate angle.

23. The mobile device of claim 17 wherein:

the plurality of instructions comprise instructions to check whether at a candidate angle, one or more attributes of projection profiles meet at least one test for presence of a straight line of pixels in the particular region.

24. The mobile device of claim 17 wherein:

the plurality of instructions are configured to be executed prior to classification of the subset of first regions as text or non-text.

25. A mobile device comprising:

a camera to capture an image of an environment outside the mobile device;

a memory coupled to the camera for storing the image;

means, coupled to the memory, for identifying a set of regions in the image;

wherein each region in the set of regions is identified based on variation of intensities of pixels;

means for selecting a subset of first regions by filtering out second regions, from among the set of regions identified in the image, at least based on one or more attributes related to geometry of the regions;

means for determining a plurality of candidate angles of skew, at least based on pixels in the a particular region in the subset of first regions;

means for identifying as an angle of skew of the image, a most common candidate angle among the plurality of candidate angles, by use of at least one processor; and

means for rotating by negative of the angle of skew of the image, at least the subset of first regions.