Mobile terminal and controlling method thereof

• US 10,104,208 B2
• Filed: 07/24/2016
• Issued: 10/16/2018
• Est. Priority Date: 12/29/2015
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Baek, Bumhyeon, Jang, Hyukjoon
• Primary Examiner(s): Bilodeau, David
• Application Number:
  US15/218,046
• Publication Number:
  US 20170187852A1
• Time in Patent Office: 814 Days
• Field of Search: None
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/632 for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

Disclosed are a mobile terminal for providing an intelligent agent (IA) service and controlling method thereof. The mobile terminal includes a display unit, a camera, a sensing unit comprising a microphone configured to sense a surrounding voice of the mobile terminal, an angle sensor configured sense an angle of the mobile terminal, and a location sensor configured to sense a location of the mobile terminal, a communication unit configured to transmit/receive data with a server configured to store a context information of a user, and a controller, if a trigger signal including a preset voice signal is sensed through the microphone, activating an intelligent agent (IA), the controller, if a preset input signal is sensed in a state that the IA is activated, activating the camera.

Claims

1. A mobile terminal, comprising:

a display;

a camera;

a sensing unit comprising a microphone;

a communication unit configured to communicate with a server; and

a controller configured to;

activate an intelligent agent (IA) in response to a trigger signal, the trigger signal comprising a preset voice signal received via the microphone;

activate the camera in response to a preset input signal sensed in a state that the IA has been activated;

sense at least one object from a preview image recognized within a view angle area of the activated camera or from an image captured via the activated camera; and

receive, via the communication unit, context information associated with the sensed at least one object from the server.

2. The mobile terminal of claim 1, wherein the controller is further configured to:

extract image context information corresponding to the preview image or the captured image;

extract voice context information corresponding to a voice received via the microphone, the voice received within a preset period of time after the trigger signal is sensed; and

provide IA information based on the extracted image context information and voice context information.

3. The mobile terminal of claim 2, wherein the controller is further configured to:

extract location context information from location information of the mobile terminal; and

provide IA information based on the extracted image context information and location context information.

4. The mobile terminal of claim 2, wherein:

the image context information corresponds to at least two preview images sensed within the view angle area of the camera; and

the at least two preview images comprise a first preview image and a second preview image.

5. The mobile terminal of claim 4, wherein the controller is further configured to:

sense the second preview image within a preset period time after sensing the first preview image; and

provide IA information based on the first preview image and the second preview image when a preset period time expires after sensing the second preview image.

6. The mobile terminal of claim 5, wherein the controller is further configured to:

extract location context information from location information of the mobile terminal; and

provide the IA information based on the location context information, the first preview image, and the second preview image.

7. The mobile terminal of claim 2, wherein:

the image context information corresponds to at least two capture images sensed within the view angle area of the camera; and

the at least two capture images comprise a first capture image and a second capture image.

8. The mobile terminal of claim 7, wherein the controller is further configured to:

extract location context information from location information of the mobile terminal; and

provide IA information based on the location context information, the first capture image, and the second capture image.

9. The mobile terminal of claim 2, wherein the image context information comprises at least one capture image sensed from the preview image within the view angle area of the camera and at least one screen shot image for content displayed via the display.

10. The mobile terminal of claim 9, wherein context information of a user is stored in the server and the controller is further configured to:

cause the communication unit to receive additional context information of the user sensed by an external device; and

provide IA information based on the received additional context information and the image context information.

11. The mobile terminal of claim 2, wherein the IA information is provided as at least one of visual information or voice information.

12. The mobile terminal of claim 2, wherein the IA information comprises tag information indicating a keyword included in the image context information or the voice context information.

13. The mobile terminal of claim 1, wherein while the IA has been activated, the controller is further configured to:

extract image context information corresponding to at least one screen shot image for content displayed via the display; and

provide IA information based on the extracted image context information.

14. The mobile terminal of claim 13, wherein the controller is further configured to cause the communication unit to transmit the IA information to the server when an IA of an external device for which user authentication is activated.

15. The mobile terminal of claim 1, wherein the controller is further configured to:

execute an IA related application when the IA is activated; and

cause the display to display a preview image sensed in a view angle area of the camera via the IA related application when the camera is activated.

16. A method for controlling a mobile terminal, the method comprising:

sensing, via a sensing unit, a trigger signal, the trigger signal comprising a preset voice signal received via a microphone of the mobile terminal;

activating, via a controller, an intelligent agent (IA) in response to the sensed trigger signal;

sensing, via the sensing unit, a preset input signal in a state that the IA has been activated;

activating, via the controller, a camera of the mobile terminal in response to the preset input signal;

sensing, via the controller, at least one object from a preview image recognized within a view angle area of the activated camera or from an image captured via the activated camera; and

receiving, via a communication unit, context information associated with the sensed at least one object from a server.

17. The method of claim 16, further comprising:

extracting image context information corresponding to the preview image or the captured image;

extracting voice context information corresponding to a voice received via the microphone within a preset period of time after the trigger signal is sensed; and

providing IA information based on the extracted image context information and voice context information.

18. The method of claim 17, further comprising:

extracting location context information from location information of the mobile terminal; and

providing IA information based on the extracted image context information and location context information.

19. A mobile terminal, comprising:

a display;

a camera;

a sensing unit comprising a microphone;

a communication unit configured to communicate with a server; and

a controller configured to;

activate an intelligent agent (IA) and the camera in response to a trigger signal and a preset input signal, respectively, the trigger signal comprising a preset voice signal received via the microphone;

cause the display to display an indicator indicating the activation of the intelligent agent;

sense at least one object from a preview image recognized within a view angle area of the activated camera or an image captured via the activated camera; and

receive, via the communication unit, context information associated with the sensed at least one object from the server.