Mobile terminal for matching displayed text with recorded external audio and method of controlling the mobile terminal

• US 10,241,743 B2
• Filed: 01/24/2014
• Issued: 03/26/2019
• Est. Priority Date: 03/05/2013
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Choe, Dami, Kim, Hyungsup, Joung, Wonseok, Park, Jungmin, Shin, Yongkyoung
• Primary Examiner(s): Shih, Haoshian
• Application Number:
  US14/771,452
• Publication Number:
  US 20160011847A1
• Time in Patent Office: 1,887 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04M 1/72433 for voice messaging, e.g. dictaphones for answering incoming calls H04M1/64
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041

Abstract

The present invention relates to a mobile terminal capable of recording audio and a method of controlling the mobile terminal. Furthermore, the present invention relates a mobile terminal capable of outputting visual information and a method of controlling the mobile terminal. Provided is the mobile terminal including a recording function unit that records external audio, a display unit to which visual information is output, and a controller that performs processing by associating the audio recorded through the recording function unit and the visual information that is output to the display unit with each other, in which in response to selection of one region of the display unit, the controller matches the visual information included in the one region and the audio recorded through the recording function unit.

Claims

1. A mobile terminal comprising:

a recording function unit that records external audio;

a touch screen; and

a controller configured to;

control the touch screen to display visual information comprising text and a graphic object for receiving a request for recording of the external audio;

perform, in response to a selection of the graphic object, the recording of the external audio in a state in which the visual information is displayed;

receive a touch input starting from the graphic object and ending at a region at which the visual information is displayed;

match a specific portion of the text displayed at the region with a specific portion of the external audio that was recorded at a point in time at which the touch input starting from the graphic object was received;

control the touch screen to highlight the display of the specific portion of the text to indicate an existence of the match of the specific portion of the text with the specific portion of the recorded external audio; and

output the specific portion of the recorded external audio when the highlighted display of the specific portion of the text is touched after the recording of the external audio is completed.

2. The mobile terminal of claim 1,

wherein the specific portion of the recorded external audio comprises audio data that has been recorded a predetermined time earlier than the point in time at which the touch input starting from the graphic object was received.

3. The mobile terminal of claim 1, wherein the graphic object comprises information output to the touch screen alerting a user that the external audio is being recorded through the recording function unit.

4. The mobile terminal of claim 3, wherein the touch input comprises a first touch input and a second touch input,

wherein the selection of the graphic object and the region is made through the first touch input with respect to the graphic object and the second touch input is in succession to the first touch input and is ended at one point corresponding to the region.

5. The mobile terminal of claim 4, wherein the graphic object is a progress bar indicating a status on progress in the recording of the external audio by the recording function unit, and

wherein when the first touch input is applied with respect to the progress bar, the controller matches the specific portion of the external audio that is recorded at the point in time corresponding to a point in time at which the first touch input is applied at the graphic object.

6. The mobile terminal of claim 5, wherein the controller converts the specific portion of the external audio that is recorded through the recording function unit into text, based on speech-to-text (SST) conversion and outputs at least one part of the resulting text close to or in a vicinity of the progress bar, and

wherein the at least one part of the text is output to a position that corresponds to a point on the progress bar that corresponds to a point in time when the specific portion of the external audio corresponding to the at least one part of the text is recorded.

7. The mobile terminal of claim 5, wherein the controller outputs an icon that alerts a user that the specific portion of the text displayed at the region is matched with a region corresponding to a point on the progress bar, to which the first touch input is applied.

8. The mobile terminal of claim 7, wherein when the icon is selected after ending the recording of the external audio by the recording function unit, the controller outputs the specific portion of the text displayed at the region, along with the specific portion of the external audio that is recorded at a point in time corresponding to the point in time at which the first touch input is applied.

9. The mobile terminal of claim 1, wherein the controller outputs an icon that alerts a user to a presence of the specific portion of the recorded external audio matched with a second region adjacent to the region.

10. The mobile terminal of claim 1, wherein an amount of the specific portion of the recorded external audio matched with the specific portion of the text displayed at the region is represented by a time from a point in time corresponding to a point in time when the region is selected to a point in time when a command to end the matching is generated.

11. A method of controlling a mobile terminal, the method comprising:

controlling a touch screen to display visual information comprising text and a graphic object for receiving a request for recording of external audio;

performing, in response to a selection of the graphic object, the recording of the external audio in a state in which the visual information is displayed;

receiving a touch input starting from the graphic object and ending at a region at which the visual information is displayed;

matching a specific portion of the text displayed at the region with a specific portion of the external audio that was recorded at a point in time at which the touch input starting from the graphic object was received;

controlling the touch screen to highlight the display of the specific portion of the text to indicate an existence of the match of the specific portion of the text with the specific portion of the recorded external audio; and

outputting the specific portion of the recorded external audio when the highlighted display of the specific portion of the text is touched after the recording of the external audio is completed.