Camera based method for text input and keyword detection

• US 8,988,543 B2
• Filed: 04/28/2011
• Issued: 03/24/2015
• Est. Priority Date: 04/30/2010
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Gktekin, Cneyt, Tenchio, Oliver
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): CHIU, WESLEY JASON
• Application Number:
  US13/095,937
• Publication Number:
  US 20110267490A1
• Time in Patent Office: 1,426 Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32048 Auto-dialling or Auto-calling H04N1/32053 - H04N1/32074 take precedence
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The present invention relates to a camera based method for text input and detection of a keyword or of a text-part within page or a screen comprising the steps of: directing a camera module on the printed page and capturing an image thereof; digital image filtering of the captured image; detection of word blocks contained in the image, each word block containing most likely a recognizable word; performing OCR within each word block; determination of A-blocks among the word blocks according to a keyword probability determination rule, wherein each of the A-blocks contains most likely the keyword; assignment of an attribute to each A-block; indication of the A-blocks in the display by a frame or the like for a further selection of the keyword; further selection of the A-block containing the keyword based on the displayed attribute of the keyword; forwarding the text content as text input to an application.

Claims

1. Camera based method for text input and detection of a keyword of a text-part within a printed page or a screen containing the keyword/text-part, the method comprising:

a) directing a camera module on the printed page or on the screen and continuously displaying the image acquired by the camera module;

b) capturing an image with the camera module, containing the keyword and text-part therein;

c) detecting a plurality of word blocks contained in the captured image, each word block containing most likely a recognizable word;

d) determining A-blocks among the plurality of word blocks according to a keyword probability determination rule, the keyword probability determination rule resulting in a respective probability value for each word block and being based at least in part on a spatial analysis of the word blocks relative to at least a portion of the captured image, the spatial analysis further being relative to an indication of a user intention of a keyword to obtain for selection, wherein each A-block contains most likely the keyword that a user of the camera module is intending to obtain for the selection from amongst the plurality of word blocks in the captured image, such that the A-blocks are preselected among the word blocks for the selection based upon each respective probability value of the A-blocks being above a threshold;

e) upon the selection of an A-block, forwarding the text content of the selected A-block as text input to a current application running on a mobile communication device containing the camera or to a camera connected electronic device.

2. The method according to claim 1, comprising displaying the captured image, and

f) digital image filtering of the captured image including contrast enhancement, shadow compensation, unwarping and rotation of the captured image in order to obtain an artifact reduced image with a substantially horizontal text alignment, wherein the detection of the plurality of word blocks is based on the artifact reduced image;

g) performing OCR within each word block to get its text content;

h) assigning to each A-block an attribute;

i) indicating the A-blocks in the display by a frame or a background color and displaying their attributes as overlays within the artifact reduced and displayed image for the selection of the keyword;

j) the selection of the A-block containing the keyword being based on the displayed attribute of the keyword; and

k) upon the selection of the A-block, displaying the text content of the selected A-block.

3. The method according to claim 2, wherein the text-part which has to be detected comprises multiple words of the displayed image, the text-part starting with a first keyword and ending with a second keyword, and wherein (j) gets executed for the selection of the first keyword and for the selection of the second keyword, and wherein (e) comprises taking the text-part and forwarding it as text input to the current application.

4. The method according to claim 2, wherein at least one of (a), (b), (i) and (j) further comprises an additional zoom function to zoom the keyword or text-part in or out for a better targeting and selection possibility.

5. The method according to claim 2, wherein the keyword probability determination within (d) takes into account for each detected word contained in the word blocks a keyword probability parameter determined by a correlation with a database, the database containing words with a low probability and words with high probability, and whereupon the word blocks containing so calculated most likely keywords are preselected as A-blocks for the selection.

6. The method according to claim 1, wherein the keyword probability determination within (d) takes into account text characteristics or text part characteristics, and whereupon the word blocks containing so calculated most likely keywords are preselected as A-blocks for the selection.

7. The method according to claim 1, wherein the keyword probability determination of (d) takes into account a statistical analysis in how often the respective word occurs within the captured image or within a section thereof, or wherein the keyword probability determination takes into account the distance of the respective word block or A-block from the center point of the displayed image.

8. The method according to claim 2, wherein at least one of (c) and (d) comprises the detection of a fingertip or of a pen-tip within the artifact reduced image indicating the keyword, whereupon the position and the direction of the fingertip or the pen-tip is taken into account for the determination of the possible keywords with their respective A-blocks;

or if there are detected two fingertips or pen-tips then the text in between is taken as text-part and forwarded as text input.

9. The method according to claim 2, wherein (j) further comprises the selection of the keyword among the predetermined indicated A-blocks via at least one of arrow keys and keypress.

10. The method according to claim 5, wherein the database for the keyword probability determination used in (d) is selectable among databases or is updatable from a connected server in regard to a use case or a currently running application for the detection of the keyword or wherein the selection of the database is accomplished within (d) according to a detected use case in regard to an analysis of the text contained in the artifact reduced image or of the words contained in the word blocks, respectively, or in regard to a detected image structure of the artifact reduced image.

11. The method according to claim 2, wherein at least one of (a), (b), (i), (j) comprise a displayed center point within the displayed image as a reticule, a cross-hair, a round or as a rectangle section whereas a size of the section is changeable;

or

wherein within (i) and (j), the detected A-blocks are indicated in the display by a highlighted background or by a highlighted text therein or by a highlighted frame in black, white or in any color;

or

wherein within (j) the selection of the assigned A-block is done by manually touching on the respective A-block within the touch-screen display, or wherein the text-part gets selected by sliding with a pen or the like along the respective text on the touch-screen display;

or wherein the selection of the assigned A-block gets done via voice recognition.

12. The method according to claim 2, wherein (e) comprises an automatic or semi-automatic correction of the text content of the selected A-block containing the keyword, before forwarding it as the text input to the current application.

13. The method according to claim 2, wherein within (d) the keyword probability determination rule takes into account a database which is related to an application or the text or structure of the artifact reduced image.

14. The method according to claim 2, wherein within (d) the keyword probability determination rule detects in particular phone numbers within the artifact reduced image as possible keywords for the selection thereof.

15. The method according to claim 2, wherein (c) comprises the detection of text-parts within the captured and artifact reduced image which get designated by attributes within step (h), whereupon within (j) one of the text-parts is selectable whereupon the text-part gets OCR converted and used as text input.

16. The method according to claim 1, wherein within (e) the detected and selected text-part is taken as first text input and sent to a server system, whereupon data related to the selected text-part get received from the server system and displayed in the display for another selection as the text input to the current running application.

17. The method according to claim 2, wherein (h) further comprises enumerating the attributes for the A-blocks starting from a center point of the displayed image and increasing up towards the margins.

18. The method according to claim 1, wherein the indication of the user intention of the keyword to obtain for selection comprises at least one of a center point of the image and an indication by a fingertip or pen tip.

19. The method according to claim 1, wherein the forwarding the text content of the selected A-block is performed upon selection of the A-block by speech recognition.