Integrated remote control device receiving multimodal input and method of the same

• US 20060077174A1
• Filed: 09/23/2005
• Published: 04/13/2006
• Est. Priority Date: 09/24/2004
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Chung, Ji-hye, Yoo, Myung-hyun, Kim, Kee-eung, Jung, Seung-nyang
• Granted Patent: US 7,538,711 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 characterized by hardware details
• CPC Class Codes: H04N 21/42221 Transmission circuitry, e.g. infrared [IR] or radio frequency [RF]
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805

Abstract

An integrated remote control device and method using multi-modal inputs are provided. The integrated remote control device may include an input module that receives inputs of at least two modalities from a user, a first modality recognition module that recognizes and interprets an input of a first modality received from the input module, a second modality recognition module that recognizes and interprets an input of a second modality received from the input module, a command database (DB) that stores commands, a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command that corresponds to the interpreted user command from the command DB, and an output module that outputs the control command extracted by the control module.

Claims

1. An integrated remote control device receiving multi-modal inputs, comprising:

an input module that receives inputs of at least two modalities from a user;

a first modality recognition module that recognizes and interprets an input of a first modality received from the input module;

a second modality recognition module that recognizes and interprets an input of a second modality received from the input module;

a command database(DB) that stores commands;

a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command that corresponds to the interpreted user command from the command DB; and

an output module that outputs the control command extracted by the control module.

2. The integrated remote control device of claim 1, wherein the commands comprise simple commands and complex control commands comprising at least some of the simple commands.

3. The integrated remote control device of claim 1, wherein the commands comprise basic commands pre-defined by a provider of the integrated remote control device, and user-defined commands.

4. The integrated remote control device of claim 1, wherein the command DB further stores correlations between controlled devices.

5. The integrated remote control device of claim 1, further comprising:

a control history DB that stores information about a state of a controlled device that is changed upon execution of the control command; and

a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in the control history DB.

6. The integrated remote control device of claim 5, wherein upon interpretation of the user command, the controlled device and a portion that controls the state of the controlled device are extracted from information that is interpreted by the first and second modality recognition modules, and if an error is detected as a result of the extraction, the error is automatically recovered from based on information stored in the control history DB or feedback informing the user of the error is provided to the user, depending on the degree of the error.

7. The integrated remote control device of claim 6, wherein the feedback is output in different modes according to a usage environment.

8. The integrated remote control device of claim 5, wherein the control command extracted by the control module is reconstructed according to the state information of a controlled device.

9. The integrated remote control device of claim 5, wherein the control command extracted by the control module is reconstructed according to both the correlation information that indicates a relationship of the controlled device to other devices, and the state information of the controlled device.

10. The integrated remote control device of claim 1, wherein the input of the first modality comprises a voice.

11. The integrated remote control device of claim 1, wherein the input of the second modality comprises a gesture.

12. The integrated remote control device of claim 1, wherein the control module interprets the command of the user, and, in a user input mode, receives a new user-defined command from the user and stores the user-defined command in the command DB.

13. The integrated remote control device of claim 1, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module automatically produces a new user-defined command corresponding to the command of the user, and a correlation between the new controlled device and other controlled devices, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

14. The integrated remote control device of claim 1, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module receives a new user-defined command and a correlation between the new controlled device and other controlled devices from the user, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

15. An integrated remote control system in which a control device remotely controls a controlled device using a control command, the control device comprising:

a command database (DB) that stores commands;

a control module that interprets a command of a user and extracts a control command corresponding to the interpreted user command from the command DB; and

an output module that outputs the control command extracted by the control module, wherein the controlled device comprises;

a receiving module that receives the control command outputted by the output module; and

wherein an input module that receives inputs of at least two modalities from the user, a first modality recognition module that recognizes and interprets an input of a first modality received from the input module, and a second modality recognition module that recognizes and interprets an input of a second modality received from the input module are included in at least one of the control device and the controlled device.

16. The integrated remote control system of claim 15, wherein the commands comprise simple commands and complex control commands that comprise at least some of the simple commands.

17. The integrated remote control system of claim 15, wherein the commands comprise basic commands pre-defined by a provider of the integrated remote control device, and user-defined commands.

18. The integrated remote control system of claim 15, wherein the command DB further stores correlations between controlled devices.

19. The integrated remote control system of claim 15, wherein at least one of the control device and the controlled device further comprises:

a control history DB that stores information about a state of the controlled device that is changed upon execution of the control command; and

a state monitoring module that checks a state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in the control history DB.

20. The integrated remote control system of claim 19, wherein upon interpretation of the user command by the control module, the controlled device and a portion that controls the state of the controlled device are extracted from the input, and if an error is detected, the error is automatically recovered from based on information stored in the control history DB or a feedback informing the user of the error is provided to the user, depending on the degree of the error.

21. The integrated remote control system of claim 20, wherein the feedback is output in different modes according to a usage environment.

22. The integrated remote control system of claim 19, wherein the control command extracted by the control module is reconstructed according to the state information of a controlled device.

23. The integrated remote control system of claim 19, wherein the control command extracted by the control module is reconstructed according to both the correlation information that details a relationship of the controlled device to other devices, and the state information of the controlled device.

24. The integrated remote control system of claim 15, wherein the input of the first modality comprises a voice.

25. The integrated remote control system of claim 15, wherein the input of the second modality comprises a gesture.

26. The integrated remote control system of claim 15, wherein the control module interprets the command of the user and, in a user input mode, receives a new user-defined command from the user and stores the user-defined command in the command DB.

27. The integrated remote control system of claim 15, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module automatically produces a new user-defined command corresponding to the command of the user and a correlation between the new controlled device and other controlled devices, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

28. The integrated remote control system of claim 15, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module receives a new user-defined command and a correlation between the new controlled device and other controlled devices from the user, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

29. An integrated remote control method receiving multi-modal inputs, comprising:

receiving inputs of at least two modalities from a user;

recognizing and interpreting each of the inputs of at least two modalities;

interpreting a command of the user by synthesizing the inputs of at least two modalities;

extracting a control command corresponding to the interpreted user command; and

outputting the extracted control command.

30. The integrated remote control method of claim 29, further comprising monitoring a state of a controlled device that is changed upon execution of the control command, and reflecting the change in the state of the controlled device in a control history DB.

31. The integrated remote control method of claim 30, wherein the interpreting of the user command comprises:

extracting the controlled device and a portion that controls the state of the controlled device from the inputs of at least two modalities; and

if an error is detected in the extracted portions, automatically recovering from the error based on information stored in the control history DB or providing feedback informing the user of the errors, according to the degree of the error.

32. The integrated remote control method of claim 31, wherein the feedback is output in different modes according to a usage environment.

33. The integrated remote control method of claim 29, wherein at least one of the inputs of the two or more modalities comprises speech.

34. The integrated remote control method of claim 29, wherein at least one of the inputs of the two or more modalities comprises a gesture.

35. The integrated remote control method of claim 29, further comprising receiving a new user-defined command from the user in a user input mode and storing the user-defined command in a command DB after the initial interpretation of the user command.

36. The integrated remote control method of claim 29, wherein the extracting of the control command comprises:

extracting a control command corresponding to the interpreted user command; and

reconstructing the control command according to state information about a controlled device indicated by the extracted control command.

37. The integrated remote control method of claim 29, wherein the extracting of the control command comprises:

extracting a control command corresponding to the interpreted user command;

extracting correlation information that details a relationship of a controlled device indicated by the extracted control command to other controlled devices; and

reconstructing the control command according to the correlation and state information of the controlled device indicated by the extracted control command.

38. The integrated remote control method of claim 29 further comprising:

automatically producing a new user-defined command corresponding to the user command and a correlation between a new controlled device and other controlled device;

authenticating the new user-defined command and the correlation; and

storing the new user-defined command and the correlation in the command DB when the new controlled device is detected as a result of the interpretation of the user command.

39. The integrated remote control method of claim 29 further comprising:

when a new controlled device is detected as a result of the interpretation of the user command, receiving a new user-defined command and a new user-defined correlation from a user;

authenticating the new user-defined command and the new user-defined correlation; and

storing the authenticated new user-defined command and new user-defined correlation in the command DB.

40. A storage for controlling a device according to an integrated remote control method receiving multi-modal inputs, comprising:

receiving inputs of at least two modalities from a user;

recognizing and interpreting each of the inputs of at least two modalities;

interpreting a command of the user by synthesizing the inputs of at least two modalities;

extracting a control command corresponding to the interpreted user command; and

outputting the extracted control command.