Mobile terminal and controlling method for displaying a written touch input based on a recognized input voice

• US 9,390,715 B2
• Filed: 05/07/2014
• Issued: 07/12/2016
• Est. Priority Date: 09/17/2013
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Jeon, Hyunjoo
• Primary Examiner(s): Pham, Tammy
• Assistant Examiner(s): Leland, III, Edwin S
• Application Number:
  US14/272,233
• Publication Number:
  US 20150081291A1
• Time in Patent Office: 797 Days
• Field of Search: 704/235
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/171 by use of digital ink
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A mobile terminal including a wireless communication unit configured to perform wireless communication; a microphone configured to receive an input voice; a touch screen; and a controller configured to receive a written touch input on the touch screen corresponding to the input voice, recognize the input voice while a voice recognition mode is activated, and display extracted information extracted from the recognized input voice on the touch screen based on a comparison of the recognized input voice and the written input.

Claims

1. A mobile terminal comprising:

a wireless communication unit configured to perform wireless communication;

a microphone configured to receive an input voice;

a touch screen; and

a controller configured to;

receive a written touch input on the touch screen corresponding to the input voice,

recognize the input voice while a voice recognition mode is activated,

display extracted information extracted from the recognized input voice on the touch screen based on a comparison of the recognized input voice and the written touch input, wherein the extracted information does not overlap the written touch input,

when the extracted information is selected, insert the extracted information into a region on which the written touch input is displayed and display next extracted information on the touch screen,

activate an automatic input mode based on a difference in speed between the written touch input and the input voice,

automatically insert the extracted information into the region on which the written touch input is displayed in response the activated automatic input mode, and

inactivate the automatic input mode when the written touch input is detected in the automatic input mode.

2. The mobile terminal of claim 1, wherein the controller is further configured to display the extracted information at a position on the touch screen corresponding to the written touch input.

3. The mobile terminal of claim 2, wherein the controller is further configured to stop displaying the written touch input when the extracted information is displayed at the position on the touch screen corresponding to the written touch input.

4. The mobile terminal of claim 1, wherein the controller is further configured to:

convert the recognized input voice into text, and

display the text on the touch screen at a first position different than a position where the written touch input is displayed.

5. The mobile terminal of claim 4, wherein the controller is further configured to display the extracted information when receiving a selection of the displayed text.

6. The mobile terminal of claim 1, wherein portions of the written touch input corresponding to the extracted information are highlighted from other portions of the extracted information.

7. The mobile terminal of claim 6, wherein the controller is further configured to automatically correct misspelling of the portions of the written touch input and automatically add remaining letters excluded from the written touch input.

8. The mobile terminal of claim 1, wherein the controller is further configured to change information displayed on the touch screen or display a new page to secure a memo space for receiving the written touch input.

9. The mobile terminal of claim 1, wherein when the written touch input includes a written picture, the controller is further configured to display an emoticon that is similar in shape to the written picture image with the extracted information displayed on the touch screen.

10. The mobile terminal of claim 1, wherein the controller is further configured to:

switch between a first screen mode and a second screen mode based on a predetermined input, and

wherein the first screen mode displays only the written touch input and the second screen mode displays both the written touch input and text of the recognized input voice.

11. The mobile terminal of claim 10, wherein the controller is further configured to:

display a playback display option for playing back a recording of the input voice, and

scroll the written touch input in the first screen mode and scroll the written touch input and the text of the recognized input voice in the second screen mode according to a speed at which the recorded voice is played back.

12. The mobile terminal of claim 1, wherein the controller is further configured to enter an edit mode and output an indicator indicating an input position to a point at which a predetermined touch input is detected.

13. The mobile terminal of claim 1, wherein when the voice recognition mode is changed to an inactivated state or recording of the voice input is finished, the controller is further configured to:

display a list of characters that can be added to the written touch input, and

add selected characters from the list of characters to the written touch input at a position corresponding to a touch and drag from the selected characters to the written touch input.

14. The mobile terminal of claim 1, wherein when an automatic input function is selected the voice recognition mode is activated.

15. The mobile terminal of claim 1, wherein when a first control command is detected when a screen of the written touch input corresponding to a first application is displayed on the touch screen, the controller is further configured to:

switch the screen of the written touch input to an execution screen corresponding to a second application, and

display a graphic object indicating that recording of the voice input is in progress on one region of the touch screen.

16. The mobile terminal of claim 15, wherein when it is detected that a predetermined-type touch input is applied to the graphic object, the controller is further configured to:

display the screen of the written touch input corresponding to the first application back on the touch screen, and

insert characters corresponding to the voice input recorded while the execution screen corresponding to the second application is displayed into a region to which the written touch input is displayed.

17. The mobile terminal of claim 1, wherein when the controller determines a speed of the written touch input applied on the touch screen is higher or lower by a predetermined range than an input speed of the voice, the controller is further configured to:

execute the automatic input mode in which the recognized input voice is converted into text, and

add the text to a region to which the written touch input is displayed.

18. The mobile terminal of claim 1, wherein the controller is further configured to:

receive a control command for executing an additional search associated with the voice input,

analyze the voice input,

search a memory associated with the mobile terminal or conduct a web search for a voice file that matches the voice input, and

display a list of found voice files on the touch screen.

19. A method of controlling a mobile terminal, the method comprising:

receiving, via a microphone, an input voice;

receive, via a touch screen, a written touch input corresponding to the input voice;

recognizing, via a controller, the input voice while a voice recognition mode is activated; and

displaying, via the controller, extracted information extracted from the recognized input voice on the touch screen based on a comparison of the recognized input voice and the written touch input, wherein the extracted information does not overlap the written touch input;

when the extracted information is selected, inserting the extracted information into a region on which the written touch input is displayed and displaying next extracted information on the touch screen;

activating an automatic input mode based on a difference in speed between the written touch input and the input voice;

automatically inserting the extracted information into the region on which the written touch input is displayed in response the activated automatic input mode; and

inactivating the automatic input mode when the written touch input is detected in the automatic input mode.

20. The method of claim 19, further comprising:

displaying the extracted information at a position on the touch screen corresponding to the written touch input.