ELECTRONIC DEVICE AND CONTROLLING METHOD THEREOF

• US 20180122379A1
• Filed: 11/03/2017
• Published: 05/03/2018
• Est. Priority Date: 11/03/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: SOHN, Young-chul, PARK, Gyu-tae, LEE, Ki-beom, LEE, Jong-ryul
• Granted Patent: US 10,679,618 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 12/28 characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks] wireless communication networks H04W ; arrangements for dividing the transmission path H04W40/00
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59

Abstract

An approach for controlling method of an electronic device is provided. The approach acquires voice information and image information for setting an action to be executed according to a condition, the voice information and the image information being respectively generated from a voice and a behavior associated with the voice of a user. The approach determines an event to be detected according to the condition and a function to be executed according to the action when the event is detected, based on the acquired voice information and the acquired image information. The approach determines at least one detection resource to detect the determined event. In response to the at least one determined detection resource detecting at least one event satisfying the condition, the approach executes the function according to the action.

Claims

1. A controlling method of an electronic device, the method comprising:

acquiring voice information and image information setting an action to be executed according to a condition, the voice information and the image information being generated from a voice and a behavior;

determining an event to be detected according to the condition and a function to be executed according to the action, based on the voice information and the image information;

determining at least one detection resource to detect the event; and

in response to the at least one detection resource detecting the event satisfying the condition, executing the function according to the action.

2. The method of claim 1, wherein the determining the at least one detection resource comprises:

retrieving pre-installed available resources; and

determining at least one detection resource, among the retrieved pre-installed available resources, to detect the event using a detection function of the at least one detection resource.

3. The method of claim 1, wherein the at least one detection resource is a module included in the electronic device or an external device positioned outside the electronic device.

4. The method of claim 1, further comprising:

in response to the at least one detection resource being determined, transmitting control information requesting detection of the event to the at least one determined detection resource.

5. The method of claim 1, further comprising:

retrieving pre-installed available resources; and

determining at least one execution resource, among the retrieved pre-installed available resources, to execute the function according to the action using an execution function of the determined at least one execution resource.

6. The method of claim 5, wherein the executing the function according to the action comprises transmitting control information to the determined at least one execution resource for the determined at least one execution resource to execute the function according to the action.

7. The method of claim 1, wherein the executing the function according to the action comprises:

receiving a result of detection of the event from the detection resource; and

executing the function according the action based on the received detection result.

8. The method of claim 1, further comprising providing, in response to there being no detection resource to detect the event or in response to the detection resource not being capable of detecting the event, a notification user interface (UI) notifying that execution of the action according to the condition is not possible.

9. The method of claim 1, wherein the determining the event to be detected comprises determining the condition and the action according to an intent of a user by applying the voice information and image information to a data recognition model generated using a learning algorithm.

10. The method of claim 9, wherein the determining the condition and the action according to the intent of the user further comprises:

providing a notification user interface (UI) for identifying the condition and the action to the user.

11. An electronic device, comprising:

a memory; and

a processor configured;

to acquire voice information and image information setting an action to be executed according to a condition, the voice information and the image information being generated from a voice and a behavior,

to determine an event to be detected according to the condition and a function to be executed according to the action, based on the voice information and the image information,

to determine at least one detection resource to detect the event, and

to execute, in response to the at least one determined detection resource detecting the event satisfying the condition, the function according to the action.

12. The electronic device of claim 11, wherein the processor is further configured:

to retrieve, in response to determining the at least one detection resource, pre-installed available resources, and

to determine at least one detection resource, among the retrieved pre-installed available resources, to detect the event using a detection function of the at least one detection resource.

13. The electronic device of claim 11, wherein the at least one detection resource is a module included in the electronic device and an external device located outside the electronic device.

14. The device of claim 11, wherein the electronic device further comprises a communicator configured to communicate with the at least one detection resource, and

wherein the processor is further configured to control, in response to the at least one detection resource being determined, the communicator to transmit control information requesting for detection of the event to the at least one determined detection resource.

15. The device of claim 11, wherein the processor is further configured:

to retrieve pre-installed available resources, and

to determine at least one execution resource, among the retrieved pre-installed available resources, to execute the function according to the action using an execution function of the determined at least one execution resource.

16. The device of claim 15, wherein the electronic device further comprises a communicator configured to communicate with the execution resource, and

wherein the processor is further configured to transmit, in response to executing the function according to the action, control information to the determined at least one execution resource for the determined at least one execution resource to execute the function according to the action.

17. The device of claim 11, wherein the processor is further configured:

to receive, in response to executing the function according to the action, a result of detection of the event from the detection resource, and

to execute the function according to the action based on the received detection result.

18. The device of claim 11, wherein the electronic device further comprises a display configured to display a user interface (UI), and

wherein the processor is further configured to control, in response to there being no detection resource to detect the event or in response to the detection resource not being capable of detecting the event, the display to display a notification UI informing that execution of the action according to the action is not possible.

19. The device of claim 11, wherein the processor is further configured to:

determine, in response to determining a function to be executed according to an event to be detected and the action according to the condition based on the voice information and the image information, the condition and action according to an intent of a user by applying the voice information and the image information to a data recognition model generated using a learning algorithm, and

determine an event to be detected according to the condition and a function to be executed according to the action.

20. The device of claim 19, wherein the electronic device further comprises a display configured to display a user interface (UI), and

wherein the processor is further configured to control, in response to determining the condition and the action according to the intent of the user, the display to display a notification UI for identifying the condition and the action to the user.