CAMERA BASED METHOD FOR TEXT INPUT AND KEYWORD DETECTION

• US 20110267490A1
• Filed: 04/28/2011
• Published: 11/03/2011
• Est. Priority Date: 04/30/2010
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Beyo GmbH
• Inventors: Göktekin, Cüneyt, Tenchio, Oliver
• Granted Patent: US 8,988,543 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/32048 Auto-dialling or Auto-calling H04N1/32053 - H04N1/32074 take precedence
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

The present invention relates to a camera based method for text input and detection of a keyword or of a text-part within page or a screen comprising the steps of: directing a camera module on the printed page and capturing an image thereof; digital image filtering of the captured image; detection of word blocks contained in the image, each word block containing most likely a recognizable word; performing OCR within each word block; determination of A-blocks among the word blocks according to a keyword probability determination rule, wherein each of the A-blocks contains most likely the keyword; assignment of an attribute to each A-block; indication of the A-blocks in the display by a frame or the like for a further selection of the keyword; further selection of the A-block containing the keyword based on the displayed attribute of the keyword; forwarding the text content as text input to an application.

Claims

1. Camera based method for text input and detection of a keyword or of a text-part within a printed page or a screen containing the keyword/text-part comprising the steps of:

a) directing a camera module on the printed page or on the screen and continuously displaying the image acquired by the camera module;

b) capturing an image with the camera module, containing the keyword or text-part therein and displaying the captured image;

c) digital image filtering of the captured image including contrast enhancement, shadow compensation, unwarping and rotation of the captured image in order to obtain an artifact reduced image with a substantially horizontal text alignment;

d) detection of word blocks contained in the artifact reduced image, each word block containing most likely a recognizable word;

e) performing OCR within each word block to get its text content;

f) determination of A-blocks among the word blocks according to a keyword probability determination rule, wherein each A-block containing most likely the keyword, such that the A-blocks are preselected among the word blocks for a further selection;

g) assignment to each A-block an attribute, which can preferably be an enumerated number;

h) indication of the A-blocks in the display by a frame or a background color and displaying their attributes as overlays within the artifact reduced and displayed image for the further selection of the keyword;

i) further selection of an A-block containing the keyword based on the displayed attribute of the keyword;

j) displaying the text content of the further selected A-block and forwarding the text content as text input to a current application running on a mobile communication device containing the camera or to a camera connected electronic device.

2. The method according to claim 1, wherein step (g) further comprises an enumeration of the attributes for the A-blocks starting from a center point of the displayed image and increasing up towards the margins.

3. The method according to claim 1, wherein the text-part which has to be detected comprises multiple words of the displayed image, the text-part starting with a first keyword and ending with a second keyword, and wherein step (i) gets executed for the selection of the first keyword and for the selection of the second keyword, and wherein step (j) comprises taking the text-part and forwarding it as text input to the current application.

4. The method according to claim 1, wherein at least one of the steps (a), (b), (h) and (i) further comprises an additional zoom function to zoom the keyword or text-part in or out for a better targeting and further selection possibility.

5. The method according to claim 1, wherein the keyword probability determination within step (f) takes into account for each detected word contained in the word blocks a keyword probability parameter determined by a correlation with a database, the database containing words with a low probability as for instance articles and filler words and words with high probability, and whereupon the word blocks containing so calculated most likely keywords are preselected as A-blocks for a further selection.

6. The method according to claim 1, wherein the keyword probability determination within step (f) takes into account text characteristics or text part characteristics, said characteristics comprising word underlining, word written in bold letters, word starting with a capital letter, hand-underlined word, word underlined with another color or the like, and whereupon the word blocks containing so calculated most likely keywords are preselected as A-blocks for a further selection.

7. The method according to claim 1, wherein the keyword probability determination of step (f) takes into account a statistical analysis in how often the respective word occurs within the captured image or within a section thereof, and or wherein the keyword probability determination takes into account the distance of the respective word block or A-block from the center point of the displayed image.

8. The method according to claim 1, wherein step (d) and/or step (f) comprises the detection of a fingertip or of a pen-tip within the artifact reduced image indicating the keyword, whereupon the position and the direction of the fingertip or the pen-tip is taken into account for the determination of the possible keywords with their respective A-blocks;

or if there are detected two fingertips or pen-tips then the text in between is taken as text-part and forwarded as text input.

9. The method according to claim 1, wherein step (i) further comprises the selection of the keyword among the predetermined indicated A-blocks, via arrow keys and keypress or the like.

10. The method according to claim 5, wherein the database for the keyword probability determination used in step (f) is selectable among databases and/or is updatable from a connected server in regards to a use case or a currently running application for the detection of the keyword and/or wherein the selection of the database is accomplished within step (f) according to a detected use case in regards to an analysis of the text contained in the artifact reduced image or of the words contained in the word blocks, respectively, and/or in regards to a detected image structure of the artifact reduced image.

11. The method according to claim 1, wherein at least one of the steps (a), (b), (h), (i) comprise a displayed center point within the displayed image as a reticule, a cross-hair, a round or as a rectangle section whereas a size of the section is changeable;

and/or

wherein within step (h) and (i) the detected A-blocks are indicated in the display by a highlighted background or by a highlighted text therein or by a highlighted frame in black, white or in any color; and

/or

wherein within step (i) the selection of the assigned A-block is done by manually touching on the respective A-block within the touch-screen display, or wherein the text-part gets selected by sliding with a pen or the like along the respective text on the touch-screen display; and

/or wherein the selection of the assigned A-block gets done via voice recognition.

12. The method according to claim 1, wherein step (j) comprises an automatic or semi-automatic correction of the text content of the further selected A-block containing the keyword, before forwarding it as the text input to the current application.

13. The method according to claim 1, wherein within step (f) the keyword probability determination rule takes into account a database which is related to an application and/or the text and/or the structure of the artifact reduced image.

14. The method according to claim 1, wherein within step (f) the keyword probability determination rule detects in particular phone numbers within the artifact reduced image as possible keywords for a further selection thereof.

15. The method according to claim 1, wherein step (d) comprises the detection of text-parts within the captured and artifact reduced image which get designated by attributes within step (g), whereupon within step (i) one of the text-parts is further selectable by the respective key, by cursor keys or the like, whereupon the text-part gets OCR converted and used as text input.

16. The method according to claim 1, wherein within step (j) the detected and further selected text-part is taken as first text input and sent to a server system, whereupon data related to the further selected text-part get received from the server system and displayed in the display for another selection as the text input to the current running application.