Integrated remote control device receiving multimodal input and method of the same
First Claim
1. An integrated remote control device receiving multi-modal inputs, the device comprising:
- an input module that receives inputs of at least two modalities from a user;
a first modality recognition module that recognizes and interprets an input of a first modality received from the input module;
a second modality recognition module that recognizes and interprets an input of a second modality received from the input module;
a command database (DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which the user operates the devices;
a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command from the command DB, the control command being for remotely controlling a plurality of devices and corresponding to the interpreted user command;
an output module that outputs the control command extracted by the control module; and
a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in a control history DB,wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices.
1 Assignment
0 Petitions
Accused Products
Abstract
An integrated remote control device and method using multi-modal inputs are provided. The integrated remote control device may include an input module that receives inputs of at least two modalities from a user, a first modality recognition module that recognizes and interprets an input of a first modality received from the input module, a second modality recognition module that recognizes and interprets an input of a second modality received from the input module, a command database (DB) that stores commands, a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command that corresponds to the interpreted user command from the command DB, and an output module that outputs the control command extracted by the control module.
-
Citations
36 Claims
-
1. An integrated remote control device receiving multi-modal inputs, the device comprising:
-
an input module that receives inputs of at least two modalities from a user; a first modality recognition module that recognizes and interprets an input of a first modality received from the input module; a second modality recognition module that recognizes and interprets an input of a second modality received from the input module; a command database (DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which the user operates the devices; a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command from the command DB, the control command being for remotely controlling a plurality of devices and corresponding to the interpreted user command; an output module that outputs the control command extracted by the control module; and a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in a control history DB, wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 36)
-
-
13. An integrated remote control system in which a control device remotely controls a controlled device using a control command, the control device comprising:
-
a command database (DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the devices; a control module that interprets a command of a user and extracts a control command corresponding to the interpreted user command from the command DB; and
an output module that outputs the control command extracted by the control module, wherein the controlled device comprises;a receiving module that receives the control command outputted by the output module; and
wherein at least one of the control device and the controlled device include,an input module that receives inputs of at least two modalities from the user, a first modality recognition module that recognizes and interprets an input of a first modality received from the input module, and a second modality recognition module that recognizes and interprets an input of a second modality received from the input module are included in at least one of the control device and the controlled device, and wherein the control module interprets the command of the user by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command from the command DB, the control command being for remotely controlling a plurality of devices and corresponding to the interpreted user command; and a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in a control history DB, wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices. - View Dependent Claims (14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)
-
-
25. An integrated remote control method receiving multi-modal inputs, the method comprising:
-
receiving by a remote control inputs of at least two modalities from a user; recognizing and interpreting each of the inputs of at least two modalities; storing commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the controlled devices; interpreting a command of the user by synthesizing the inputs of at least two modalities; extracting a control command for remotely controlling a plurality of devices, the control command corresponding to the interpreted user command; checking a state of the controlled device, monitoring the state of the controlled device upon execution of the control command, and reflecting any changes in the state of the controlled device in a control history DB, wherein, when the inputs of the at least two modalities are insufficient to interpret a particular control command due to error, the particular control command is further interpreted using the state information from the control history DB and using the correlation information that indicates a relationship of the controlled device to other controlled devices; and outputting by the remote control the extracted control command. - View Dependent Claims (26, 27, 28, 29, 30, 31, 32, 33)
-
-
34. A storage for controlling a device according to an integrated remote control method receiving multi-modal inputs, the method comprising:
-
receiving by a remote control inputs of at least two modalities from a user; recognizing and interpreting each of the inputs of at least two modalities; storing commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the controlled devices; interpreting a command of the user by synthesizing the inputs of the at least two modalities; extracting a control command for remotely controlling a plurality of devices, the control command corresponding to the interpreted user command; checking a state of the controlled device, monitoring the state of the controlled device upon execution of the control command, and reflecting any changes in the state of the controlled device in a control history DB, wherein, when the inputs of the at least two modalities are insufficient to interpret a particular control command due to error, the particular control command is further interpreted using the state information from the control history DB and using the correlation information that indicates a relationship of the controlled device to other controlled devices; and outputting by the remote control the extracted control command.
-
-
35. An integrated remote control device receiving multi-modal inputs, the device comprising:
-
an input module that receives inputs of at least two modalities from a user; a first modality recognition module that recognizes and interprets an input of a first modality received from the input module; a second modality recognition module that recognizes and interprets an input of a second modality received from the input module;
a command database(DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which a user operates the devices;a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command that corresponds to the interpreted user command from the command DB; a control history DB that stores information about a state of a controlled device that is changed upon execution of the control command; and a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in the control history DB, wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices.
-
Specification