Integrated remote control device receiving multimodal input and method of the same

• US 7,538,711 B2
• Filed: 09/23/2005
• Issued: 05/26/2009
• Est. Priority Date: 09/24/2004
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jung, Seung-nyang, Chung, Ji-hye, Yoo, Myung-hyun, Kim, Kee-eung
• Primary Examiner(s): Lee; Benjamin C
• Assistant Examiner(s): Yacob; Sisay
• Application Number:
  US11/233,114
• Publication Number:
  US 20060077174A1
• Time in Patent Office: 1,341 Days
• Field of Search: 341/176, 341/174, 341/173, 348/14.01, 348/601, 348/725, 348/563, 348/734, 704/275, 704/264, 704/270, 704/270.1, 700/19, 700/17, 700/83, 345/156, 345/158, 345/163, 345/169, 701/36, 701/49, 340/825.25, 340/825.69, 340/825.72
• US Class Current: 341/176
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: H04N 21/4131 home appliance, e.g. lighting, air conditioning system, metering devices home automation data switching networks exchanging configuration information on appliance services H04L12/2807
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42206 characterized by hardware details
• CPC Class Codes: H04N 21/42221 Transmission circuitry, e.g. infrared [IR] or radio frequency [RF]
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805

Abstract

An integrated remote control device and method using multi-modal inputs are provided. The integrated remote control device may include an input module that receives inputs of at least two modalities from a user, a first modality recognition module that recognizes and interprets an input of a first modality received from the input module, a second modality recognition module that recognizes and interprets an input of a second modality received from the input module, a command database (DB) that stores commands, a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command that corresponds to the interpreted user command from the command DB, and an output module that outputs the control command extracted by the control module.

Claims

1. An integrated remote control device receiving multi-modal inputs, the device comprising:

an input module that receives inputs of at least two modalities from a user;

a first modality recognition module that recognizes and interprets an input of a first modality received from the input module;

a second modality recognition module that recognizes and interprets an input of a second modality received from the input module;

a command database (DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which the user operates the devices;

a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command from the command DB, the control command being for remotely controlling a plurality of devices and corresponding to the interpreted user command;

an output module that outputs the control command extracted by the control module; and

a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in a control history DB,

wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices.

2. The integrated remote control device of claim 1, wherein the commands comprise simple commands and complex control commands comprising at least some of the simple commands.

3. The integrated remote control device of claim 1, wherein the commands comprise basic commands pre-defined by a provider of the integrated remote control device, and user-defined commands.

4. The integrated remote control device of claim 1, wherein the control history DB further stores information about the state of the controlled device that is changed upon execution of the control command.

5. The integrated remote control device of claim 4, wherein upon interpretation of the user command, the controlled device and a portion that controls the state of the controlled device are extracted from information that is interpreted by the first and second modality recognition modules, and if an error is detected as a result of the extraction, the error is automatically recovered from based on information stored in the control history DB or feedback informing the user of the error is provided to the user, depending on the degree of the error.

6. The integrated remote control device of claim 5, wherein the feedback is output in different modes according to a usage environment.

7. The integrated remote control device of claim 4, wherein the control command extracted by the control module is reconstructed according to the state information of a controlled device.

8. The integrated remote control device of claim 1, wherein the input of the first modality comprises a voice.

9. The integrated remote control device of claim 1, wherein the input of the second modality comprises a gesture.

10. The integrated remote control device of claim 1, wherein the control module interprets the command of the user, and, in a user input mode, receives a new user-defined command from the user and stores the user-defined command in the command DB.

11. The integrated remote control device of claim 1, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module automatically produces a new user-defined command corresponding to the command of the user, and a correlation between the new controlled device and other controlled devices, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

12. The integrated remote control device of claim 1, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module receives a new user-defined command and a correlation between the new controlled device and other controlled devices from the user, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

36. the intergrated remote control device of claim 1, wherein the control command can be extractable according to characteristics of the plurality of devices.

13. An integrated remote control system in which a control device remotely controls a controlled device using a control command, the control device comprising:

a command database (DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the devices;

a control module that interprets a command of a user and extracts a control command corresponding to the interpreted user command from the command DB; and

an output module that outputs the control command extracted by the control module, wherein the controlled device comprises;

a receiving module that receives the control command outputted by the output module; and

wherein at least one of the control device and the controlled device include,

an input module that receives inputs of at least two modalities from the user,

a first modality recognition module that recognizes and interprets an input of a first modality received from the input module, and

a second modality recognition module that recognizes and interprets an input of a second modality received from the input module are included in at least one of the control device and the controlled device, and wherein

the control module interprets the command of the user by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command from the command DB, the control command being for remotely controlling a plurality of devices and corresponding to the interpreted user command;

and a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in a control history DB, wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices.

14. The integrated remote control system of claim 13, wherein the commands comprise simple commands and complex control commands that comprise at least some of the simple commands.

15. The integrated remote control system of claim 13, wherein the commands comprise basic commands pre-defined by a provider of the integrated remote control device, and user-defined commands.

16. The integrated remote control system of claim 13, wherein at least one of the control device and the controlled device further comprises the control history DB and where the control history DB stores information about the state of the controlled device that is changed upon execution of the control command.

17. The integrated remote control system of claim 16, wherein upon interpretation of the user command by the control module, the controlled device and a portion that controls the state of the controlled device are extracted from the input, and if an error is detected, the error is automatically recovered from based on information stored in the control history DB or a feedback informing the user of the error is provided to the user, depending on the degree of the error.

18. The integrated remote control system of claim 17, wherein the feedback is output in different modes according to a usage environment.

19. The integrated remote control system of claim 16, wherein the control command extracted by the control module is reconstructed according to the state information of a controlled device.

20. The integrated remote control system of claim 13, wherein the input of the first modality comprises a voice.

21. The integrated remote control system of claim 13, wherein the input of the second modality comprises a gesture.

22. The integrated remote control system of claim 13, wherein the control module interprets the command of the user and, in a user input mode, receives a new user-defined command from the user and stores the user-defined command in the command DB.

23. The integrated remote control system of claim 13, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module automatically produces a new user-defined command corresponding to the command of the user and a correlation between the new controlled device and other controlled devices, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

24. The integrated remote control system of claim 13, wherein when the control module detects a new controlled device by interpreting a command of the user, the control module receives a new user-defined command and a correlation between the new controlled device and other controlled devices from the user, authenticates the new user-defined command and the correlation, and stores the new user-defined command and the correlation in the command DB.

25. An integrated remote control method receiving multi-modal inputs, the method comprising:

receiving by a remote control inputs of at least two modalities from a user;

recognizing and interpreting each of the inputs of at least two modalities;

storing commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the controlled devices;

interpreting a command of the user by synthesizing the inputs of at least two modalities;

extracting a control command for remotely controlling a plurality of devices, the control command corresponding to the interpreted user command;

checking a state of the controlled device, monitoring the state of the controlled device upon execution of the control command, and reflecting any changes in the state of the controlled device in a control history DB,

wherein, when the inputs of the at least two modalities are insufficient to interpret a particular control command due to error, the particular control command is further interpreted using the state information from the control history DB and using the correlation information that indicates a relationship of the controlled device to other controlled devices; and

outputting by the remote control the extracted control command.

26. The integrated remote control method of claim 25, wherein the interpreting of the user command comprises:

extracting the controlled device and a portion that controls the state of the controlled device from the inputs of at least two modalities; and

if an error is detected in the extracted portions, automatically recovering from the error based on information stored in the control history DB or providing feedback informing the user of the errors, according to the degree of the error.

27. The integrated remote control method of claim 26, wherein a feedback to the user is output in different modes according to a usage environment.

28. The integrated remote control method of claim 25, wherein at least one of the inputs of the two or more modalities comprises speech.

29. The integrated remote control method of claim 25, wherein at least one of the inputs of the two or more modalities comprises a gesture.

30. The integrated remote control method of claim 25, further comprising receiving a new user-defined command from the user in a user input mode and storing the user-defined command in a command DB after the initial interpretation of the user command.

31. The integrated remote control method of claim 25, wherein the extracting of the control command comprises:

extracting a control command corresponding to the interpreted user command; and

reconstructing the control command according to state information about a controlled device indicated by the extracted control command.

32. The integrated remote control method of claim 25 further comprising:

automatically producing a new user-defined command corresponding to the user command and a correlation between a new controlled device and other controlled device;

authenticating the new user-defined command and the correlation; and

storing the new user-defined command and the correlation in the command DB when the new controlled device is detected as a result of the interpretation of the user command.

33. The integrated remote control method of claim 25 further comprising:

when a new controlled device is detected as a result of the interpretation of the user command, receiving a new user-defined command and a new user-defined correlation from a user;

authenticating the new user-defined command and the new user-defined correlation; and

storing the authenticated new user-defined command and new user-defined correlation in the command DB.

34. A storage for controlling a device according to an integrated remote control method receiving multi-modal inputs, the method comprising:

receiving by a remote control inputs of at least two modalities from a user;

recognizing and interpreting each of the inputs of at least two modalities;

storing commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the controlled devices;

interpreting a command of the user by synthesizing the inputs of the at least two modalities;

extracting a control command for remotely controlling a plurality of devices, the control command corresponding to the interpreted user command;

checking a state of the controlled device, monitoring the state of the controlled device upon execution of the control command, and reflecting any changes in the state of the controlled device in a control history DB,

wherein, when the inputs of the at least two modalities are insufficient to interpret a particular control command due to error, the particular control command is further interpreted using the state information from the control history DB and using the correlation information that indicates a relationship of the controlled device to other controlled devices; and

outputting by the remote control the extracted control command.

35. An integrated remote control device receiving multi-modal inputs, the device comprising:

an input module that receives inputs of at least two modalities from a user;

a first modality recognition module that recognizes and interprets an input of a first modality received from the input module;

a second modality recognition module that recognizes and interprets an input of a second modality received from the input module;

a command database(DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the devices;

a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command that corresponds to the interpreted user command from the command DB;

a control history DB that stores information about a state of a controlled device that is changed upon execution of the control command; and

a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in the control history DB,

wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices.