Multimodal control device and multimodal control method

• US 7,657,502 B2
• Filed: 09/13/2005
• Issued: 02/02/2010
• Est. Priority Date: 05/13/2005
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Matsumoto, Tatsuro, Sasaki, Kazuo, Yano, Ai
• Primary Examiner(s): Pham; Khanh B
• Application Number:
  US11/224,033
• Publication Number:
  US 20060259450A1
• Time in Patent Office: 1,603 Days
• Field of Search: 715/520
• US Class Current: 715/248
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: H04M 2203/251 where a voice mode or a visual mode can be used interchangeably
• CPC Class Codes: H04M 3/4938 comprising a voice browser which renders and interprets, e.g. VoiceXML
• CPC Class Codes: H04M 7/006 Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42

Abstract

To provide a multimodal control device and method, wherein existing browsers can be used to achieve synchronization with fine granularity, while making full use of GUI browsability. A multimodal control device comprises input focus managing means for managing input focus items in document data, content generation means for generating content, in accordance with a modality type, from document data within a data input item range including input focus items, and content transmission means for transmitting generated content to the corresponding browser.

Claims

1. A multimodal control device comprising for each modality type, a browser for interpreting content in accordance with modality type and performing interaction with a user, wherein content containing an input focus item currently subject to input is generated from document data having a plurality of data input items, which are stored in a storage unit, said content being within a data input item range in accordance with the characteristics of each modality, and output to each browser, the multimodal control device comprising:

a computer executing

moving input currently subject to input in the document data and managing the moved input item currently subject to input as an input focus item, in response to receiving a processing request from the browser;

referring to a content granularity definition file which stores a correspondency between the modality types and the data input item range of content to be generated in the modality;

specifying a data input item range containing the input focus item from the document data, according to the referred content granularity definition file;

generating content containing the specified data input item range; and

transmitting the generated content containing the specified data input item range to a browser corresponding to the modality to display the generated content.

2. The multimodal control device according to claim 1, wherein the input focus managing changes input focus item in response to detection of input of data in a data input item from any of the modalities.

3. The multimodal control device according to claim 2, wherein the content generating generates content to confirm whether to direct transmission of the input data to the corresponding browser with respect to content for a modality having a data input item range of 1, after completion of data input in the plurality of data input items.

4. The multimodal control device according to claim 3, wherein the content generating generates the content which includes the inputted data as transmission data, for transmission confirmation.

5. The multimodal control device according to claim 1, wherein the browsers include a browser for GUI modality, wherein the content generating generates a content for the browser for GUI modality, which includes a content clearly indicating a data input item corresponding to an input focus location.

6. The multimodal control device according to claim 5, wherein the content generating generates a content for the browser for GUI modality, which includes interim input results data embedded in an entry form as initial values.

7. The multimodal control device according to claim 1, wherein the content generating, when content for each browser is generated from document data comprising a plurality of input items, generates a character string unique to each data input item and generates the content which includes the generated character string as an input identifier common to all modalities.

8. The multimodal control device according to claim 1, wherein the document data comprises a definition file in which content corresponding to modality type is defined for each data input item, wherein the content generating, in accordance with modality type and input focus item, reads content for each modality corresponding to the relevant data input item from the definition file.

9. The multimodal control device according to claim 1, wherein the content generating generates the content which is embedded a program for notifying data of results of interim input into the data input items.

10. The multimodal control device according to claim 1, wherein the content generating generates the content which is embedded a program for detecting movement of input focus and notifying input focus item move information.

11. A multimodal control device comprising for each modality type, a browser for interpreting content generated from document data in accordance with modality type, the multimodal control device comprising:

a computer executing

moving input currently subject to input in the document data and managing the moved input item currently subject to input as an input focus item, in response to receiving a processing request from the browser;

referring to a content granularity definition file which stores a correspondency between the modality types and the data input item range of content to be generated in the modality;

specifying a data input range containing the input focus item from the document data, according to the referred content granularity definition file;

generating content containing the specified data input item range; and

transmitting the generated content containing the specified data input item range to a browser corresponding to the modality to display the generated content data.

12. A multimodal control method where for each modality type, a browser interprets content in accordance with modality type and interacts with a user, wherein content containing an input focus item currently subject to input is generated from document data having a plurality of data input items which are stored in a storage unit, said content being within a data input item range in accordance with the characteristics of each modality, and output to each browser, the multimodal control method comprising:

storing in a computer readable recording medium a content granularity definition file which stores a correspondency between the modality types and the data input item range of content to be generated in the modality by a computer processor; and

executing by the computer processor operations of;

moving input currently subject to input in the document data and managing the moved input item currently subject to input as an input focus item, in response to receiving a processing request from the browser;

referring to the content granularity definition file which stores the correspondency between the modality types and the data input item range of content to be generated by the computer in the modality;

specifying a data input item range containing the input focus item from the document data, according to the referred content granularity definition file;

generating content containing the specified data input item range; and

transmitting the generated content containing the specified data input item range to a browser corresponding to the modality to display the generated data.

13. A computer readable recording medium storing a program causing a computer to execute a multimodal control where for each modality type, a browser is provided for interpreting content in accordance with modality type and performing interaction with a user, wherein content containing an input focus item currently subject to input is generated and output in each browser from document data having a plurality of data input items, said content being within a data input item range in accordance with the characteristics of each modality, the computer readable recording medium that stores the program causes the computer to execute the multimodal control by:

moving input currently subject to input in the document data and managing the moved input item currently subject to input as an input focus item, in response to receiving a processing request from the browser;

referring to a content granularity definition file which stores a correspondency between the modality types and the data input item range of content to be generated in the modality;

specifying a data input item range containing the input focus item from the document data, according to the referred content granularity definition file;

generating content containing the specified data input item range; and

transmitting the generated content to a browser corresponding to the modality to display the generated content.