Apparatus and method for transforming application for multi-modal interface

• US 8,881,001 B2
• Filed: 11/20/2007
• Issued: 11/04/2014
• Est. Priority Date: 11/21/2006
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Ku, Tai-Yeon, Park, Dong-Hwan, Son, Young-Sung, Moon, Kyeong-Deok, Park, Jun-Hee
• Primary Examiner(s): TAPP, AMELIA L
• Application Number:
  US11/942,861
• Publication Number:
  US 20080120447A1
• Time in Patent Office: 2,541 Days
• Field of Search: 715/238
• US Class Current: 715/238
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

Provided are an apparatus and method for transforming an application for a multi-modal interface. The apparatus includes: a parsing means for parsing an original application to generate a parsed structure of the original application; an application transforming/creating means for transforming the parsed structure generated by the parsing means; a multi-modal synthesizing means for synthesizing the contents created by the application transforming/creating means with multi-modal information of the user terminal; a multi-modal detecting means for detecting multi-modal information received from the user terminal and transforming the detected information into contents that can be used by the device; and an action mapping means for mapping the contents, which are received from the multi-modal detecting means and can be used in the device, to action information contained in the original application to generate an event of the device according to the action information.

Claims

1. A multi-modal application transform apparatus connected to a multi-modal user terminal to manage an original application of a device in a home network, the multi-modal application transform apparatus comprising:

a parsing means for receiving the original application of the device, and

wherein the parsing means parses the received original application of the device to generate a parsed structure of the original application;

an application transforming/creating means for transforming the parsed structure generated by the parsing means according to output characteristics of a user terminal to create contents that can be used by the user terminal,

wherein the user terminal is any type of a wired or wireless computer device comprising but not limited to at least one of a person computer (PC), person digital assistance (PDA), cellular phone, or watch;

a multi-modal synthesizing means for synthesizing the contents created by the application transforming/creating means with multi-modal information of the user terminal;

a multi-modal detecting means for detecting the multi-modal information received from the user terminal and transforming the detected information into contents that can be used by the device; and

an action mapping means for mapping the contents, which are received from the multi-modal detecting means and can be used in the device, to action information contained in the original application to generate an event of the device according to the action information,

wherein the received original application having the action information to generate the event of the device is received by the parsing means

wherein the parsing means automatically transforms the original application based on the output characteristics of the respective user terminal by one application writing process in order to have the action information in the original application to generate the event of the device, and

wherein the one application writing process automatically transforms the original application in real time based on each and every one of the output characteristics of at least two or more different user terminals.

2. The multi-modal application transform apparatus of claim 1, wherein the action mapping means includes:

a content transforming unit for transforming multi-modal information detected by the multi-modal detecting means into contents that can be used by the device and extracting action information of the original application corresponding to the multi-modal information;

a scoring unit for giving, using the multi-modal information detected by the multi-modal detecting means, a score to the action information extracted by the content transforming unit on the basis of the current state; and

an action transforming unit for transforming, according to the score result received from the scoring unit, the action information into multi-modal information that can be detected by the device, and providing the multi-modal information to the device.

3. The multi-modal application transform apparatus of claim 1, wherein the application transforming/creating means pre-stores and uses a policy describing a transform of the original application according to the output characteristics of the user terminal.

4. The multi-modal application transform apparatus of claim 1, wherein the application transforming/creating means creates contents that can be used by the user terminal in a text format.

5. The multi-modal application transform apparatus of claim 1, wherein the multi-modal detecting means transforms contents that can be used by the device into a text format.

6. A multi-modal application transform method for transforming a multi-modal application provided to a multi-modal user terminal to manage an original application of a device in a home network, the multi-modal application transform method comprising the steps of:

a) receiving the original application and then transforming the original application of the device according to output characteristics of a user terminal to create contents that can be used by the user terminal,

wherein the user terminal is any type of a wired or wireless computer device comprising but not limited to at least one of a person computer (PC), person digital assistance (PDA), cellular phone, or watch;

b) synthesizing the contents that can be used by the user terminal with multi-modal information of the user terminal and providing the resulting data to the user terminal;

c) transforming the multi-modal information detected by the user terminal into contents that can be used by the device and extracting action information of the original application corresponding to the multi-modal information;

d) giving, according to the multi-modal information detected by the user terminal, a score to the action information extracted from the original application corresponding to the multi-modal information; and

e) transforming, according to the score given to the action information, the action information into multi-modal information that can be detected by the device, and providing the multi-modal information to the device,

wherein the received original application having the action information transformed into the multi-modal information that can be detected by the device is received,

wherein the parsing means automatically transforms the original application based on the output characteristics of the respective user terminal by one application writing process in order to have the action information in the original application be transformed to the multi-modal information that can be detected by the device, and

wherein the one application writing process automatically transforms the original application in real time based on each and every one of the output characteristics of at least two or more different user terminals.

7. The multi-modal application transform method of claim 6, wherein the contents that can be used by the user terminal are created in a text format in the step a).

8. The multi-modal application transform method of claim 6, wherein the multi-modal information detected by the user terminal is transformed into text contents in the step c).

9. A multi-modal application transform apparatus connected to a multi-modal user terminal to manage an original application of a device in a home network, the multi-modal application transform apparatus comprising:

a parsing means for receiving the original application of the respective device, and

wherein the parsing means parses the received original application of the device to generate a parsed structure of the original application;

an application transforming/creating means for transforming the parsed structure generated by the parsing means according to output characteristics of a user terminal to create contents used by the user terminal,

wherein the user terminal is any type of a wired or wireless computer device comprising but not limited to at least one of a person computer (PC), person digital assistance (PDA), cellular phone, or watch;

a multi-modal synthesizing means for synthesizing the contents created by the application transforming/creating means with multi-modal information of the user terminal;

a multi-modal detecting means for detecting the multi-modal information received from the user terminal and transforming the detected information into contents that can be used by the device; and

an action mapping means for mapping the contents, which are received from the multi-modal detecting means and can be used in the device, to action information contained in the original application to generate an event of the device according to the action information,

wherein the received original application having the action information to generate the event of the device is received by the parsing means based on the output characteristics of the user terminal by one application writing process in order to have the action information in the original application to generate the event of the device,

wherein the multi-modal application transform apparatus receives the original application having the action information of the original device in order to have the user terminal create content to generate the event of the device, and

wherein the one application writing process automatically transforms the original application in real time based on each and every one of the output characteristics of at least two or more different user terminals.