Multimodel text input by a keyboard/camera text input module replacing a conventional keyboard text input module on a mobile device

• US 10,078,376 B2
• Filed: 06/28/2017
• Issued: 09/18/2018
• Est. Priority Date: 03/06/2012
• Status: Active Grant

• Current Assignee: Cuneyt Goktekin
• Original Assignee: Cuneyt Goktekin
• Inventors: Goktekin, Cuneyt
• Primary Examiner(s): Reed, Stephen T
• Application Number:
  US15/636,189
• Publication Number:
  US 20170300128A1
• Time in Patent Office: 447 Days
• Field of Search: None
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0227 Cooperation and interconnection of the input arrangement with other functional units of a computer G06F3/023 - G06F3/037 take precedence
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

Methods and modules for a multimodal text input in a mobile device are provided. Text may be input via keyboard or camera mode by holding the camera over written text. An image is taken of the written text, text is recognized, and output to an application by: activating a keyboard mode; providing an A-Z-keyboard in a first input field; activating the camera mode; capturing the text image and displaying the captured image in a second field of a device display; converting the captured image to character text by OCR and displaying the recognized character text on the display; outputting a selected character as the input text to the application upon a character selection, or outputting a selected part of the recognized character text as the input text to the application upon a selection of the part of the recognized character text via by a single keypress, control command, or gesture.

Claims

1. A method for multimodal text input in a mobile device comprising a touch screen display, a touch screen A-Z keyboard and a camera, the method comprising:

using an original communication interface between a third-party application and an original keyboard module, wherein the original keyboard module is replaced by a multimodal text input module communicating with said third-party application instead of the original keyboard module;

said third-party application requesting an input text through a text input field;

executing the multimodal text input module by running the multimodal text input module in the background of the mobile device and constantly monitoring the background of the mobile device to detect when the text input field of the third-party application is activated;

wherein the multimodal text input module simultaneously activates both a keyboard mode and a camera mode,

said keyboard mode configured to;

display an A-Z keyboard in a first field of the touch screen display for a character selection for the text input,

sense for a manual selection of one or more selected characters of the A-Z keyboard and outputting the selected characters to the text input field of the third-party application; and

said camera mode configured to;

capture an image of a written text by the camera;

display said captured image in a second field of the display;

convert said captured image to recognized character text by optical character recognition (OCR);

display the recognized character text as an overlaid text over the original text within the captured image, and

sense for a manual selection of a part of the displayed recognized character text and outputting said selected part of the recognized character text to the text input field of the third-party application;

wherein said capturing, displaying of said captured image, converting, and displaying of said recognized character text is capable of being accomplished by a single operation comprising a single keypress, a control command or a single gesture.

2. The method according to claim 1, wherein one or more suggestion candidates of said recognized character text are determined by an algorithm in connection with a data base and are displayed in one or more third fields or as another overlay within said second field, and said one or more suggestion candidates are selectable by a keypress event on a key, wherein said key is one or more of a group consisting of

a visible key:

and

a hidden key within one or more of a group consisting of

one or more of the fields of said display,

on the recognized character text, and

on another text on said display.

3. The method according to claim 2, wherein said keypress event providing said selection is from the group consisting of a single touch keypress and a single swipe gesture on a touch screen.

4. The method according to claim 1, wherein a size of the first field and/or the second field is the same size of the original keyboard module on said mobile device.

5. The method according to claim 1, wherein said second field is displayed adjacent to said A-Z keyboard in said first field.

6. The method according to claim 1, wherein the steps of capturing, displaying said captured image, converting, and displaying said recognized character text are executed repetitively until a certain keypress is detected to end said repetitive execution,

wherein the respective latest recognized text in the part of the respective latest captured image is analyzed for new text in regards to a previously recognized character text that is output to said application requesting said input text, whereupon a control command is generated for the selection of the new text as the recognized character text, and said new text is output to said application requesting said input text.

7. The method according to claim 6, wherein said control command for the selection of said recognized character text is generated automatically via a detection algorithm, wherein said detection algorithm recognizes whether said recognized character text in a previously captured image and in the current captured image are the same.

8. The method according to claim 1, wherein:

said keyboard mode for character selection is executed by a keyboard text input module;

said camera mode is executed by a camera text input module,

wherein the execution of said keyboard text input module is independent from the execution of said camera text input module, but

wherein the execution of said camera text input module is dependent on the execution of said keyboard text input module;

wherein said camera text input module is always active in the background to detect whether said keyboard text input module is active,

wherein detection of execution of said keyboard text input module activates said camera mode,

wherein at least the second field and the recognized character text are visible on the display; and

wherein said camera mode is deactivated if said camera text input module detects that said keyboard text input module ceases to be active.

9. A mobile device configured to facilitate multimodal text input for a third-party application, the mobile device comprising:

a display being a touch screen display configured to display text and image content in a frame;

an A-Z keyboard being a touch screen A-Z keyboard on the touch screen display having a keyboard mode, wherein said touch screen A-Z keyboard is configured to receive input text;

a camera having a camera mode; and

a processor in communication with said touch screen display, said touch screen A-Z keyboard, and said camera, said processor being configured to;

execute the third-party application using an original communication interface to an original keyboard module, wherein a multimodal text input module replaces the original keyboard module communicating with said third-party application instead of the original keyboard module, said third-party application requesting an input text through a text input field;

execute the multimodal text input module by running the multimodal text input module in the background of the mobile device and constantly monitoring the background of the mobile device to detect when the text input field of the third-party application is activated, the multimodal text input module simultaneously activates both a keyboard mode and a camera mode;

wherein said keyboard mode is configured to;

display the A-Z keyboard in a first field of the touch screen display for a character selection for text input, and

sense for a manual selection of one or more characters on the A-Z keyboard and output the selected characters to the text input field of the third-party application; and

wherein said camera mode is configured to;

capture an image of a written text by the camera;

display said captured image in a second field of the display;

convert said captured image to recognized character text by optical character recognition (OCR);

display the recognized character text as an overlaid text over the original text within the captured image;

sense for a manual selection of a part of the displayed recognized character text; and

output said selected part of the recognized character text to the text input field of the third-party application requesting the input text;

wherein said capturing, displaying of said captured image, converting, and displaying of said recognized character text is accomplished by a single operation comprising a single keypress, a control command, or a single gesture.

10. The mobile device according to claim 9, wherein said keyboard mode and said camera mode are both active simultaneously.

11. The mobile device according to claim 10, wherein said processor configures said A-Z keyboard and said camera such that they are active at the same time.

12. The mobile device according to claim 10, wherein said single operation such as said keypress, said control command, and said single gesture include the selection of said recognized character text as input text;

the selection of said character key from the A-Z keyboard as input text; and

the immediate output of said input text to said processor.

13. The mobile device according to claim 9, wherein said A-Z keyboard comprises two sub-modules, wherein a first standard keyboard sub-module is activated by the processor in response to a request for input text by said application;

and said second camera sub-module configured to be always active in the background detecting whether said first keyboard sub-module is activated.

14. The mobile device according to claim 13, wherein said processor is configured to activate said camera mode in response to the activation of said second camera sub-module, such that said second field is displayed adjacent to the A-Z keyboard;

and shut down said camera mode when said second keyboard sub-module is no longer activated by said application.