Speech Enabled Media Sharing In A Multimodal Application

• US 20110010180A1
• Filed: 07/09/2009
• Published: 01/13/2011
• Est. Priority Date: 07/09/2009
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Agapi, Clprian, Cross, Charles W. JR., Bodin, William K.
• Granted Patent: US 8,510,117 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 16/81 Indexing, e.g. XML tags; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Speech enabled media sharing in a multimodal application including parsing, by a multimodal browser, one or more markup documents of a multimodal application; identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser; loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources; receiving, by the multimodal browser, an utterance matching a keyword for the web resource, a keyword for a mode of resource sharing and a keyword for a target for receipt of the web resource in the web resource sharing grammar thereby identifying the web resource, a mode of resource sharing, and a target for receipt of the web resource; and sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing.

Claims

1. A method of speech enabled media sharing in a multimodal application, the method implemented with the multimodal application and a multimodal browser, a module of automated computing machinery operating on a multimodal device supporting multiple modes of user interaction, the modes of user interaction including a voice mode and one or more non-voice modes, wherein the voice mode includes accepting speech input from a user, digitizing the speech, and providing digitized speech to a speech engine, and wherein the non-voice mode includes accepting input from a user through physical user interaction with a user input device for the multimodal device;

the method comprising;

parsing, by the multimodal browser, one or more markup documents of a multimodal application;

identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser;

loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources;

receiving, by the multimodal browser, an utterance matching a keyword for the web resource, a keyword for a mode of resource sharing and a keyword for a target for receipt of the web resource in the web resource sharing grammar thereby identifying the web resource, a mode of resource sharing, and a target for receipt of the web resource; and

sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing.

2. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprises:

identifying more than one web resource for simultaneous display in the multimodal browser; and

providing, for each web resource for simultaneous display in the multimodal browser, a unique visual designation.

3. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprises

identifying more than one web resource for simultaneous display in the multimodal browser;

identifying metadata associated with each web resource for simultaneous display in the multimodal browser; and

loading a disambiguating grammar including keywords selected in dependence upon the metadata associated with each web resource.

4. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprises identifying an element in the multimodal application containing metadata describing the location of the web resource.

5. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprises identifying markup designating a hyperlink to a web resource.

6. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprises identifying a file type consistent with a web resource.

7. The method of claim 1 wherein sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing further comprises sending a reference to the web resource.

8. An apparatus of speech enabled media sharing in a multimodal application, the apparatus including a multimodal application and a multimodal browser, a module of automated computing machinery operating on a multimodal device supporting multiple modes of user interaction, the modes of user interaction including a voice mode and one or more non-voice modes, wherein the voice mode includes accepting speech input from a user, digitizing the speech, and providing digitized speech to a speech engine, and wherein the non-voice mode includes accepting input from a user through physical user interaction with a user input device for the multimodal device;

the apparatus comprising computer program instructions for

parsing, by the multimodal browser, one or more markup documents of a multimodal application;

identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser;

loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources;

receiving, by the multimodal browser, an utterance matching a keyword for the web resource, a keyword for a mode of resource sharing and a keyword for a target for receipt of the web resource in the web resource sharing grammar thereby identifying the web resource, a mode of resource sharing, and a target for receipt of the web resource; and

sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing.

9. The apparatus of claim 8 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for:

identifying more than one web resource for simultaneous display in the multimodal browser; and

providing, for each web resource for simultaneous display in the multimodal browser, a unique visual designation.

10. The apparatus of claim 8 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for

identifying more than one web resource for simultaneous display in the multimodal browser;

identifying metadata associated with each web resource for simultaneous display in the multimodal browser; and

loading a disambiguating grammar including keywords selected in dependence upon the metadata associated with each web resource.

11. The apparatus of claim 8 wherein computer program instructions identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprises computer program instructions identifying an element in the multimodal application containing metadata describing the location of the web resource.

12. The apparatus of claim 8 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for identifying markup designating a hyperlink to a web resource.

13. The apparatus of claim 8 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for identifying a file type consistent with a web resource.

14. The apparatus of claim 8 wherein computer program instructions for sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing further comprise computer program instructions for sending a reference to the web resource.

15. An computer program product of speech enabled media sharing in a multimodal application, the computer program product including a multimodal browser, a module of automated computing machinery operating on a multimodal device supporting multiple modes of user interaction, the modes of user interaction including a voice mode and one or more non-voice modes, wherein the voice mode includes accepting speech input from a user, digitizing the speech, and providing digitized speech to a speech engine, and wherein the non-voice mode includes accepting input from a user through physical user interaction with a user input device for the multimodal device;

the computer program product comprising computer program instructions for;

parsing, by the multimodal browser, one or more markup documents of a multimodal application;

identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser;

loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources;

receiving, by the multimodal browser, an utterance matching a keyword for the web resource, a keyword for a mode of resource sharing and a keyword for a target for receipt of the web resource in the web resource sharing grammar thereby identifying the web resource, a mode of resource sharing, and a target for receipt of the web resource; and

sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing.

16. The computer program product of claim 15 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for:

identifying more than one web resource for simultaneous display in the multimodal browser; and

providing, for each web resource for simultaneous display in the multimodal browser, a unique visual designation.

17. The computer program product of claim 15 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for

identifying more than one web resource for simultaneous display in the multimodal browser;

identifying metadata associated with each web resource for simultaneous display in the multimodal browser; and

loading a disambiguating grammar including keywords selected in dependence upon the metadata associated with each web resource.

18. The computer program product of claim 15 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for identifying markup designating a hyperlink to a web resource.

19. The computer program product of claim 15 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser further comprise computer program instructions for identifying a file type consistent with a web resource.

20. The computer program product of claim 15 wherein computer program instructions for sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing further comprise computer program instructions for sending a reference to the web resource.