Method and system for detecting and recognizing text in images

• US 8,009,928 B1
• Filed: 09/19/2008
• Issued: 08/30/2011
• Est. Priority Date: 01/23/2008
• Status: Expired due to Fees

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Manmatha, Raghavan, Ruzon, Mark A.
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): Entezari; Michelle
• Application Number:
  US12/284,283
• Time in Patent Office: 1,075 Days
• Field of Search: 382/176
• US Class Current: 382/283
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/15 Cutting or merging image elements, e.g. region growing, watershed or clustering-based techniques
• CPC Class Codes: G06V 30/153 using recognition of characters or words
• CPC Class Codes: G06V 30/155 Removing patterns interfering with the pattern to be recognised, such as ruled lines or underlines
• CPC Class Codes: G06V 30/162 Quantising the image signal
• CPC Class Codes: G06V 30/164 Noise filtering
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks

Abstract

Various embodiments of the present invention relate to a method, system and computer program product for detecting and recognizing text in the images captured by cameras and scanners. First, a series of image-processing techniques is applied to detect text regions in the image. Subsequently, the detected text regions pass through different processing stages that reduce blurring and the negative effects of variable lighting. This results in the creation of multiple images that are versions of the same text region. Some of these multiple versions are sent to a character-recognition system. The resulting texts from each of the versions of the image sent to the character-recognition system are then combined to a single result, wherein the single result is detected text.

Claims

1. A computer-implemented method for detecting and recognizing text in an image, the method comprising:

under the control of one or more computer systems configured with executable instructions,

processing an input image to form an output image by filtering and segmenting the input image and intersecting the filtered and segmented input image with a mask created from a plurality of bounding boxes, each bounding box enclosing a connected component, each connected component including a plurality of pixels comprising the image and connected on the basis of a predetermined pixel intensity and predefined distance between the pixels, the output image comprising text regions detected by said filtering, segmenting, and intersecting;

separately processing the input image to create at least one binary chip, each binary chip comprising one detected text region;

recognizing the text in each binary chip from the detected text region comprising the binary chip using an optical character recognizer;

separately and independently recognizing the text from the text regions comprising the output image resulting from processing the input image using the optical character recognizer; and

logically combining the separately recognized text by taking the logical OR of the recognized text to form a single output producing detected text.

2. The computer-implemented method of claim 1, wherein processing an image to form the output image and the processing of the image to create the at least one binary chip comprises:

converting the image to a gray-level version of the image to form a gray-level image;

filtering and thresholding the gray-level image to emphasize bar-like and line-like regions which potentially correspond to text to form a first image;

removing long straight lines and rectangles from the first image to form a second image;

identifying potential text regions in the second image based on predefined character constraints to form a third image;

removing the potential text regions from the third image which are not consistent with words to create a fourth image;

creating the output image and at least one gray-level image chip from the fourth image; and

processing each of the at least one gray-level image chips to form the at least one binary chip.

3. The computer-implemented method of claim 2, wherein filtering and thresholding the gray-level image comprises:

filtering the gray-level image using a Laplacian-of-Gaussian filter to emphasize bar-like and line-like regions which potentially correspond to the text present in the gray-level image; and

thresholding the filtered gray-level image using an algorithm selected from an adaptive thresholding algorithm and a constant thresholding algorithm to form the first image.

4. The computer-implemented method of claim 2, wherein removing long straight lines and rectangles from the first image comprises:

computing connected components in the first image;

computing a ratio of the perimeter to a maximum distance transformed for the connected components, the maximum distance transformed for the connected components comprising a maximum dimension of a bounding box for each connected component; and

removing the connected components having the ratio of the perimeter to the maximum distance transformed for the connected components larger than a threshold value to remove long straight lines and rectangles, to form the second image.

5. The computer-implemented method of claim 2, wherein identifying potential text regions in the second image based on predefined character constraints comprises:

computing connected components in the second image;

checking if the connected components satisfy at least one of predefined character constraints; and

removing the connected components not satisfying at least one of the predefined character constraints to identify potential text regions to create the third image.

6. The computer-implemented method of claim 5, wherein the predefined character constraints are at least one of:

spacing between characters, ratio of height and width of adjacent characters, or vertical alignment of characters.

7. The computer-implemented method of claim 2, wherein removing the potential text regions from the third image not consistent with words comprises:

a) performing an image-closing operation on the third image with a predefined structuring element;

b) computing connected components in the third image after the image-closing operation;

c) checking if the connected components satisfy at least one of the predefined word constraints; and

d) removing the connected components not satisfying at least one of the predefined word constraints to create the fourth image.

8. The computer-implemented method of claim 7, wherein the predefined structuring element is chosen so that it connects the characters corresponding to words.

9. The computer-implemented method of claim 7, wherein the predefined word constraints are at least one of:

aspect ratio, height and width of a bounding box of the connected components of the third image, or the number of pixels in the connected components divided by width and height of the bounding box of the connected components in the third image.

10. The computer-implemented method of claim 2, wherein creating the output image and the at least one gray-level image chip comprises:

computing connected components in the fourth image;

finding a bounding box for each of the connected components present in the fourth image;

computing the intersection of the first image and the union of the bounding boxes from the fourth image to create the output image; and

extracting, for each bounding box, the corresponding image area from the gray-level image to form at least one gray-level image chip.

11. The computer-implemented method of claim 2, wherein processing the at least one gray-level image chip to form the at least one binary chip comprises:

equalizing a histogram of each of the at least one gray-level image chips; and

thresholding each of the at least one histogram-equalized gray-level image chips to form the at least one binary chip.

12. The computer-implemented method of claim 1, wherein combining the recognized text comprises taking a majority vote.

13. A system for detecting and recognizing text in an image, the system comprising:

a text-detection module, the text-detection module configured to detect text regions in an image by filtering and segmenting the image and intersecting the filtered and segmented image with a mask created from a plurality of bounding boxes, each bounding box enclosing a connected component, each connected component including a plurality of pixels comprising the image and connected on the basis of a predetermined pixel intensity and predefined distance between the pixels to form an output image and to form at least one gray-level image chip;

a chip-processing module, the chip-processing module configured to process the at least one gray-level image chip to form at least one binary chip; and

an optical character-recognizer module, the optical character-recognizer module configured to;

recognize the text in the at least one binary chip using an optical character-recognizer;

separately and independently recognize the text in the output image using the optical character-recognizer; and

logically combining the separately recognized text by taking the logical OR of the recognized text to form a single output producing detected text.

14. The system of claim 13, wherein the text-detection module comprises:

a gray-level conversion module, the gray-level conversion module converting the image to a gray-level version of the image to form a gray-level image;

a filtering and thresholding module, the filtering and thresholding module filtering and thresholding the gray-level image to emphasize bar-like and line-like regions which potentially correspond to the text, to form a first image;

a line-removal module, the line-removal module removing long straight lines and rectangles from the first image to form a second image;

a character-finding module, the character-finding module identifying potential text regions in the second image based on predefined character constraints to form a third image;

a word-finding module, the word-finding module removing potential text regions from the third image which are not consistent with words to create a fourth image; and

an output text-image-creation module, the output text-image-creation module creating the output image and the at least one gray-level image chip from the fourth image.

15. The system of claim 14, wherein the filtering and thresholding module comprises:

a filtering module, the filtering module filtering the gray-level image using a Laplacian-of-Gaussian filter to emphasize bar-like and line-like regions which potentially correspond to the text present in the gray-level image; and

a thresholding module, the thresholding module thresholding the filtered gray-level image using an algorithm selected from an adaptive thresholding algorithm and a constant thresholding algorithm to form the first image.

16. The system of claim 14, wherein the line-removal module comprises:

a connected component module, the connected component module computing connected components in the first image;

a line constraint-detection module, the line constraint-detection module computing a ratio of the perimeter to a maximum distance transformed for the connected components, the maximum distance transformed for the connected components comprising a maximum dimension of a bounding box for each connected component; and

a line-removal processor, the line-removal processor removing the connected components having the ratio of the perimeter to the maximum distance transformed for the connected components larger than a threshold value to remove long straight lines and rectangles to form the second image.

17. The system of claim 14, wherein the character-finding module comprises:

a connected component module, the connected component module computing connected components in the second image;

a character constraint-detection module, the character constraint-detection module checking if the connected components satisfy at least one of the predefined character constraints; and

a non-text removing processor, the non-text removing processor removing the connected components not satisfying at least one of the predefined character constraints to identify potential text regions to create the third image.

18. The system of claim 14, wherein the word-finding module comprises:

an image-closing module, the image-closing module performing an image-closing operation on the third image with a predefined structuring element;

a connected component module, the connected component module computing connected components in the third image after the image-closing operation;

a text word constraints-detection module, the text word constraints-detection module checking if the connected components satisfy at least one of the predefined word constraints; and

a non-text words removing processor, the non-text words removing processor removing the connected components not satisfying at least one of the predefined word constraints to create the fourth image.

19. The system of claim 14, wherein the output text-image-creation module comprises:

a connected component module, the connected component module computing connected components in the fourth image;

a bounding box module, the bounding box module finding a bounding box for each of the connected components present in the fourth image;

an output image module, the output image module computing the intersection of the first image and the union of the bounding boxes from the fourth image to create the output image; and

a gray-level chip module, the gray-level chip module extracting, for each bounding box, the corresponding image area from the gray-level image to form the at least one gray-level image chip.

20. The system of claim 13, wherein the chip-processing module comprises:

a histogram-equalizing module, the histogram-equalizing module equalizing a histogram of each of the at least one gray-level image chips; and

a thresholding module, the thresholding module thresholding each of the at least one histogram-equalized gray-level image chips to form the at least one binary chip.

21. A non-transitory computer program product for use with a computer, the computer program product comprising a computer usable medium having a computer-readable program code embodied therein for detecting and recognizing text in an image, the computer program product performing:

processing an image to form an output image by filtering and segmenting the image and intersecting the filtered and segmented image with a mask created from a plurality of bounding boxes, each bounding box enclosing a connected component, each connected component including a plurality of pixels comprising the image and connected on the basis of a predetermined pixel intensity and predefined distance between the pixels, the output image comprising text regions detected by said filtering, segmenting, and intersecting;

processing the image to create at least one binary chip, each of the at least one binary chips comprising one detected text region;

recognizing the text in each of the at least one binary chips using an optical character recognizer;

separately and independently recognizing the text from the text regions comprising the output image resulting from processing the input image using the optical character recognizer; and

logically combining the separately recognized text by taking the logical OR of the recognized text to form a single output producing detected text.

22. The non-transitory computer program product of claim 21 wherein processing the image to form the output image and processing the image to create the at least one binary chip comprises the computer program product performing:

converting the image to a gray-level version of the image to form a gray-level image;

filtering and thresholding the gray-level image to emphasize bar-like and line-like regions which potentially correspond to text to form a first image;

removing long straight lines and rectangles from the first image to form a second image;

identifying potential text regions in the second image based on predefined character constraints to form a third image;

removing the potential text regions from the third image which are not consistent with words to create a fourth image;

creating the output image and at least one gray-level image chip from the fourth image; and

processing each of the at least one gray-level image chips to form the at least one binary chip.

23. The non-transitory computer program product of claim 22, wherein filtering and thresholding the gray-level image comprises the computer program product performing:

filtering the gray-level image using a Laplacian-of-Gaussian filter to emphasize bar-like and line-like regions which potentially correspond to the text present in the gray-level image; and

thresholding the filtered gray-level image using an algorithm selected from an adaptive thresholding algorithm and a constant thresholding algorithm to form the first image.

24. The non-transitory computer program product of claim 22, wherein removing long straight lines and rectangles from the first image comprises the computer program product performing:

computing connected components in the first image;

computing a ratio of the perimeter to a maximum distance transformed for the connected components, the maximum distance transformed for the connected components comprising a maximum dimension of a bounding box for each connected component; and

removing the connected components having the ratio of the perimeter to the maximum distance transformed for the connected components larger than a threshold value to remove long straight lines and rectangles, to form the second image.

25. The non-transitory computer program product of claim 22, wherein identifying potential text regions in the second image based on predefined character constraints comprises the computer program product performing:

computing connected components in the second image;

checking if the connected components satisfy at least one of the predefined character constraints; and

removing the connected components not satisfying at least one of the predefined character constraints, to identify potential text regions to create the third image.