Identifying regions of text to merge in a natural image or video frame

• US 9,053,361 B2
• Filed: 01/23/2013
• Issued: 06/09/2015
• Est. Priority Date: 01/26/2012
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Baheti, Pawan Kumar, Gore, Dhananjay Ashok, Koo, Hyung-Il, Lee, Te-Won
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US13/748,539
• Publication Number:
  US 20130195315A1
• Time in Patent Office: 867 Days
• Field of Search: 382/100, 382/112, 382/173, 382176-187, 382/190, 382/194, 382/195, 382/202, 382/203, 382/205, 358/462, 348/135
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/18095 Summing image-intensity values; Projection and histogram analysis
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables

Abstract

In several aspects of described embodiments, an electronic device and method use a camera to capture an image or a frame of video of an environment outside the electronic device followed by identification of blocks of regions in the image. Each block that contains a region is checked, as to whether a test for presence of a line of pixels is met. When the test is met for a block, that block is identified as pixel-line-present. Pixel-line-present blocks are used to identify blocks that are adjacent. One or more adjacent block(s) may be merged with a pixel-line-present block when one or more rules are found to be satisfied, resulting in a merged block. The merged block is then subject to the above-described test, to verify presence of a line of pixels therein, and when the test is satisfied the merged block is processed normally, e.g. classified as text or non-text.

Claims

1. A method of identifying regions to merge, in an image of a scene of real world captured by a camera in a handheld device, the method of identifying regions comprising:

checking whether a first block, which contains a first region of pixels that are contiguous with one another and comprising a local extrema of intensity in the image, satisfies a predetermined test, for presence along a line, of pixels with intensities binarizable to a common value;

marking the first block as pixel-line-present, in a memory, when a result of the checking indicates the predetermined test is satisfied;

identifying a second block that is located in the image adjacent to the first block, wherein at least the first block is marked as pixel-line-present;

merging a first set of positions indicative of the first region of pixels in the first block with a second set of positions indicative of a second region of pixels in the second block to obtain a merged set of positions in a merged block, when a predetermined rule is satisfied by one or more geometric attributes of the first block and the second block;

wherein the first region of pixels and the second region of pixels do not contact one another in the merged block;

wherein the merging is performed prior to classification of any pixel in the first region of pixels and in the second region of pixels as text or non-text; and

re-doing the checking, on the merged block, to determine whether the merged block satisfies the predetermined test;

wherein one or more of the checking, the marking, the identifying, the merging, and the re-doing are performed by at least one processor coupled to the memory.

2. The method of claim 1 wherein:

the predetermined test uses at least one peak in a profile obtained by projection of intensities of pixels along a direction, wherein a property of the at least one peak satisfies a predetermined condition identified in the predetermined test.

3. The method of claim 2 wherein:

the checking comprises binarizing a plurality of pixels, within a row in a group of rows in the first block, by assigning thereto one of two values, based on comparison of intensities of said plurality of pixels with a threshold determined by use of intensities of multiple pixels in the first block; and

the checking further comprises counting within said row, a number of pixels having said common value, to identify the at least one peak in the profile.

4. The method of claim 2 wherein:

the property is a location of the at least one peak, within the profile; and

the predetermined condition compares with a predetermined constant, a ratio of the location to a span of the profile.

5. The method of claim 2 wherein:

the property is a maximum value of the profile; and

the predetermined condition compares the maximum value with a mean of multiple values in the profile.

6. The method of claim 1 further comprising:

identifying, an additional block located adjacent to the merged block; and

merging the merged set of positions and an additional set of positions in the additional block, when any rule in a plurality of predetermined rules comprising said predetermined rule is satisfied by the additional block and the merged block.

7. The method of claim 6 wherein:

the additional block is obtained by merging a third region in a third block with a fourth region in a fourth block.

8. The method of claim 1 further comprising:

prior to the merging, checking whether an additional rule is satisfied by a predetermined geometric attribute of at least one of the first block and the second block;

wherein the merging is performed in response to finding that at least said additional rule is satisfied.

9. The method of claim 1 further comprising:

prior to the checking, selecting the predetermined rule based at least on a language associated in a table with a geographic location of the scene of real world.

10. The method of claim 1 further comprising:

prior to the checking, selecting the predetermined rule based at least on a language identified by user input.

11. The method of claim 1 wherein:

a region Q_i in the image is determined by comparisons between an intensity i used as a threshold and intensities of a plurality of pixels included in said region Q_i;

said region Q_i comprises a local extrema in the intensity i in said image; and

a number of pixels in said region Q_i remains within a predetermined range relative to changes in the intensity i across a range i−

Δ

to i+Δ

, with a local minima in a ratio [Q<sub>i−

Δ</sub>

−

Q_i+Δ

]/Q_i occurring at the intensity i.

12. The method of claim 1 wherein:

the at least one processor, the memory, and the camera are comprised in the handheld device; and

the image is comprised in a video frame among a stream of video frames generated by the camera.

13. The method of claim 1 wherein:

the at least one processor, the memory, and the camera are comprised in the handheld device; and

the image is comprised in a set of snapshots captured by the camera.

14. The method of claim 1 wherein:

the at least one processor, the memory, and the camera are comprised in the handheld device; and

the image is comprised in the memory.

15. The method of claim 1 further comprising:

classifying the merged block as text or non-text depending on the result of said re-doing.

16. A mobile device comprising:

a camera;

a memory operatively connected to the camera to receive at least an image therefrom;

a processor operatively connected to the memory to execute a plurality of computer instructions stored in the memory, to supply information related to a merged block, the merged block being obtained by merging a first block with a second block;

wherein the processor is configured to check whether the first block, which contains a first region of pixels that are contiguous with one another and comprising a local extrema of intensity in the image, satisfies a predetermined test, for presence along a line, of pixels with intensities binarizable to a common value;

wherein the processor is configured to mark the first block as pixel-line-present, in the memory, when a result of the check indicates the predetermined test is satisfied;

wherein the processor is configured to identify the second block for being located in the image adjacent to the first block, when at least the first block is marked as pixel-line-present;

wherein the processor is configured to merge a first set of positions indicative of the first region of pixels in the first block with a second set of positions indicative of a second region of pixels in the second block to obtain a merged set of positions in the merged block, when a predetermined rule is satisfied by one or more geometric attributes of the first block and the second block;

wherein the first region of pixels and the second region of pixels do not contact one another in the merged block;

wherein the processor is configured to obtain the merged block prior to classification of any pixel in the first region of pixels and in the second region of pixels as text or non-text; and

wherein the processor is configured to re-do the check with the merged block, to determine whether the merged block satisfies the predetermined test.

17. The mobile device of claim 16 wherein:

the predetermined test uses at least one peak in a profile obtained by projection of intensities of pixels along a direction, wherein a property of the at least one peak satisfies a predetermined condition identified in the predetermined test.

18. The mobile device of claim 17 wherein the processor is programmed to:

binarize a plurality of pixels, within a row among a group of rows in the first block, by assigning thereto one of two values, based on comparison of intensities of said plurality of pixels with a threshold determined by use of intensities of multiple pixels in the first block; and

count within the row, a number of pixels having said common value, to identify the at least one peak in the profile.

19. The mobile device of claim 17 wherein:

the property is a location of the at least one peak, within the profile; and

the predetermined condition compares with a predetermined constant, a ratio of the location to a span of the profile.

20. The mobile device of claim 17 wherein:

the property is a maximum value of the profile; and

the predetermined condition compares the maximum value with a mean of multiple values in the profile.

21. The mobile device of claim 16 wherein the processor is programmed to:

identify, a third block located adjacent to the merged block; and

merge the merged set of positions and a third region in the third block, when any rule in a plurality of predetermined rules including said predetermined rule, is satisfied by the third block and the merged block.

22. One or more non-transitory computer readable storage media comprising computer instructions, which when executed in a handheld device, cause one or more processors in the handheld device to perform operations, the computer instructions comprising:

first instructions to check whether a first block, which contains a first region of pixels that are contiguous with one another and comprising a local extrema of intensity in an image captured by a camera, satisfies a predetermined test, for presence along a line, of pixels with intensities binarizable to a common value;

second instructions to mark the first block as pixel-line-present, in a memory, when a result of the checking indicates the predetermined test is satisfied;

third instructions to identify a second block that is located in the image adjacent to the first block, wherein at least the first block is marked as pixel-line-present;

fourth instructions to merge a first set of positions indicative of the first region of pixels in the first block with a second set of positions indicative of a second region of pixels in the second block to obtain a merged set of positions in a merged block, when a predetermined rule is satisfied by one or more geometric attributes of the first block and the second block;

wherein the first region of pixels and the second region of pixels do not contact one another in the merged block;

wherein the merging is performed prior to classification of any pixel in the first region of pixels and in the second region of pixels as text or non-text; and

fifth instructions to re-do the check with the merged block, to determine whether the merged block satisfies the predetermined test;

wherein one or more of the first instructions, the second instructions, the third instructions, the fourth instructions and the fifth instructions are to be executed by at least one processor coupled to the memory.

23. The one or more non-transitory computer readable storage media of claim 22 wherein:

the predetermined test uses at least one peak in a profile obtained by projection of intensities of pixels along a direction, wherein a property of the at least one peak satisfies a predetermined condition identified in the predetermined test.

24. The one or more non-transitory computer readable storage media of claim 23 further comprising:

sixth instructions to binarize a plurality of pixels, within a row among a group of rows in the first block, by assigning thereto one of two values, based on comparison of intensities of said plurality of pixels with a threshold determined by use of intensities of multiple pixels in the first block; and

seventh instructions to count within the row, a number of pixels having said common value, to identify the at least one peak in the profile.

25. The one or more non-transitory computer readable storage media of claim 23 wherein:

the property is a location of the at least one peak, within the profile; and

the predetermined condition compares with a predetermined constant, a ratio of the location to a span of the profile.

26. The one or more non-transitory computer readable storage media of claim 23 wherein:

the property is a maximum value of the profile; and

the predetermined condition compares the maximum value with a mean of multiple values in the profile.

27. The one or more non-transitory computer readable storage media of claim 22 wherein:

a third block located adjacent to the merged block is identified on repeated execution of the third instructions; and

the merged set of positions is further merged with a third region in the third block, when any rule in a plurality of predetermined rules including said predetermined rule is found to be satisfied by the third block and the merged block.

28. An apparatus for identifying regions of text, the apparatus comprising:

a memory storing an image of an environment outside the apparatus;

means for checking whether a first block, which contains a first region of pixels that are contiguous with one another and comprising a local extrema of intensity in the image, satisfies a predetermined test, for presence along a line, of pixels with intensities binarizable to a common value;

means for marking the first block as pixel-line-present, in the memory, when a result of the checking indicates the predetermined test is satisfied;

means for identifying a second block that is located in the image adjacent to the first block, wherein at least the first block is marked as pixel-line-present;

means for merging a first set of positions indicative of the first region of pixels in the first block with a second set of positions indicative of a second region of pixels in the second block to obtain a merged set of positions in a merged block, when a predetermined rule is satisfied by one or more geometric attributes of the first block and the second block;

wherein the first region of pixels and the second region of pixels do not contact one another in the merged block;

wherein the merging is performed prior to classification of any pixel in the first region of pixels and in the second region of pixels as text or non-text; and

means for re-doing the checking with the merged block, to determine whether the merged block satisfies the predetermined test;

wherein one or more of the means for checking, the means for marking, the means for identifying, the means for merging, and the means for re-doing comprise at least one processor coupled to the memory.

29. The apparatus of claim 28 wherein:

the predetermined test uses at least one peak in a profile obtained by projection of intensities of pixels along a direction, wherein a property of the at least one peak satisfies a predetermined condition identified in the predetermined test.