Speech enabled media sharing in a multimodal application

• US 8,510,117 B2
• Filed: 07/09/2009
• Issued: 08/13/2013
• Est. Priority Date: 07/09/2009
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Agapi, Ciprian, Bodin, William K., Cross, Charles W. Jr.
• Primary Examiner(s): Borsetti, Greg
• Application Number:
  US12/500,029
• Publication Number:
  US 20110010180A1
• Time in Patent Office: 1,496 Days
• Field of Search: None
• US Class Current: 704/275
• CPC Class Codes: G06F 16/81 Indexing, e.g. XML tags; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Speech enabled media sharing in a multimodal application including parsing, by a multimodal browser, one or more markup documents of a multimodal application; identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser; loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources; receiving, by the multimodal browser, an utterance matching a keyword for the web resource, a keyword for a mode of resource sharing and a keyword for a target for receipt of the web resource in the web resource sharing grammar thereby identifying the web resource, a mode of resource sharing, and a target for receipt of the web resource; and sending, by the multimodal browser, the web resource to the identified target for the web resource using the identified mode of resource sharing.

Claims

1. A method of speech enabled media sharing in a multimodal application, the method implemented with the multimodal application and a multimodal browser, a module of automated computing machinery operating on a multimodal device supporting multiple modes of user interaction, the modes of user interaction including a voice mode and one or more non-voice modes, wherein the voice mode includes accepting speech input from a user, digitizing the speech, and providing digitized speech to a speech engine, and wherein the non-voice mode includes accepting input from a user through physical user interaction with a user input device for the multimodal device;

the method comprising;

parsing, by the multimodal browser, one or more markup documents of a multimodal application;

identifying, by the multimodal browser, in the one or more markup documents web resources for display in the multimodal browser;

identifying metadata associated with each of the web resources identified in the one or more markup documents for simultaneous display in the multimodal browser;

loading a disambiguating grammar including keywords selected in dependence upon the metadata associated with each of the web resources;

loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources;

receiving, by the multimodal browser, a plurality of utterances matching keywords of the disambiguating grammar for at least two of the web resources, a keyword for a mode of resource sharing and a keyword for a target for receipt of the at least two of the web resources; and

sending, by the multimodal browser, the web resources to the identified target for the web resources using the identified mode of resource sharing.

2. The method of claim 1 further comprising:

parsing, by the multimodal browser, a second set of one or more markup documents;

identifying, in the second set of markup documents, a second plurality of web resources for simultaneous display in the multimodal browser;

providing, for each of the second plurality of web resources identified in the second set of markup documents for simultaneous display in the multimodal browser, a unique visual designation;

receiving, by the multimodal browser, a second plurality of utterances matching keywords for the unique visual designations of at least two of the second plurality of web resources identified in the second set of markup documents, a keyword for a mode of resource sharing and a keyword for a target for receipt of the at least two of the plurality of web resources identified in the second set of markup documents; and

sending, by the multimodal browser, the at least two of the plurality of web resources identified in the second set of markup documents to the identified target using the identified mode of resource sharing.

3. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for at least one of the web resources, an element in the multimodal application containing metadata describing the location of the web resource.

4. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for at least one of the web resources, markup designating a hyperlink to web resource.

5. The method of claim 1 wherein identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for at least one of the web resources, a file type consistent with a web resource.

6. The method of claim 1 wherein sending, by the multimodal browser, the web resources to the identified target for the web resources using the identified mode of resource sharing further comprises sending, for each of the web resources, a reference to the web resource.

7. An apparatus of speech enabled media sharing in a multimodal application, the apparatus including a multimodal application and a multimodal browser, a module of automated computing machinery operating on a multimodal device supporting multiple modes of user interaction, the modes of user interaction including a voice mode and one or more non-voice modes, wherein the voice mode includes accepting speech input from a user, digitizing the speech, and providing digitized speech to a speech engine, and wherein the non-voice mode includes accepting input from a user through physical user interaction with a user input device for the multimodal device;

the apparatus comprising computer program instructions for

parsing, by the multimodal browser, one or more markup documents of a multimodal application;

identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser;

identifying metadata associated with each of the web resources identified in the one or more markup documents for simultaneous display in the multimodal browser;

loading a disambiguating grammar including keywords selected in dependence upon the metadata associated with each of the web resources;

loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources;

receiving, by the multimodal browser, a plurality of utterances matching keywords of the disambiguating grammar for at least two of the web resources, a keyword for a mode of resource sharing and a keyword for a target for receipt of the at least two of the web resources; and

sending, by the multimodal browser, the web resources to the identified target for the web resources using the identified mode of resource sharing.

8. The apparatus of claim 7 further comprising computer program instructions:

parsing, by the multimodal browser, a second set of one or more markup documents;

identifying, in the second set of markup documents, a second plurality of web resources for simultaneous display in the multimodal browser;

providing, for each of the second plurality of web resources identified in the second set of markup documents for simultaneous display in the multimodal browser, a unique visual designation;

receiving, by the multimodal browser, a second plurality of utterances matching keywords for the unique visual designations of at least two of the second plurality of web resources identified in the second set of markup documents, a keyword for a mode of resource sharing and a keyword for a target for receipt of the at least two of the plurality of web resources identified in the second set of markup documents; and

sending, by the multimodal browser, the at least two of the plurality of web resources identified in the second set of markup documents to the identified target using the identified mode of resource sharing.

9. The apparatus of claim 7 wherein computer program instructions identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for at least one of the web resources, an element in the multimodal application containing metadata describing the location of the web resource.

10. The apparatus of claim 7 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for at least one of the web resources, markup designating a hyperlink to the web resource.

11. The apparatus of claim 7 wherein computer program instructions for identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for at least one of the web resources, a file type consistent with a web resource.

12. The apparatus of claim 7 wherein computer program instructions for sending, by the multimodal browser, the web resources to the identified target for the web resources using the identified mode of resource sharing further comprises sending, for each of the web resources, a reference to the web resource.

13. A computer program product of speech enabled media sharing in a multimodal application, computer program product comprising computer program instructions stored on a non-transitory machine-readable medium, the computer program instructions instructing a processor to perform:

for;

parsing, by the multimodal browser, one or more markup documents of a multimodal application;

identifying, by the multimodal browser, in the one or more markup documents a web resource for display in the multimodal browser;

identifying metadata associated with each of the web resources identified in the one or more markup documents for simultaneous display in the multimodal browser;

loading a disambiguating grammar including keywords selected in dependence upon the metadata associated with each of the web resources;

loading, by the multimodal browser, a web resource sharing grammar that includes keywords for modes of resource sharing and keywords for targets for receipt of web resources;

receiving, by the multimodal browser, a plurality of utterances matching keywords of the disambiguating grammar for at least two of the web resources, a keyword for a mode of resource sharing and a keyword for a target for receipt of the at least two of the web resources; and

sending, by the multimodal browser, the web resources to the identified target for the web resources using the identified mode of resource sharing.

14. The computer program product of claim 13 further comprising computer program instructions to perform:

parsing, by the multimodal browser, a second set of one or more markup documents;

identifying, in the second set of markup documents, a second plurality of web resources for simultaneous display in the multimodal browser;

providing, for each of the second plurality of web resources identified in the second set of markup documents for simultaneous display in the multimodal browser, a unique visual designation;

receiving, by the multimodal browser, a second plurality of utterances matching keywords for the unique visual designations of at least two of the second plurality of web resources identified in the second set of markup documents, a keyword for a mode of resource sharing and a keyword for a target for receipt of the at least two of the plurality of web resources identified in the second set of markup documents; and

sending, by the multimodal browser, the at least two of the plurality of web resources identified in the second set of markup documents to the identified target using the identified mode of resource sharing.

15. The computer program product of claim 13 wherein computer program instructions to perform identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for a least one of the web resources, markup designating a hyperlink to the web resource.

16. The computer program product of claim 13 wherein computer program instructions to perform identifying, by the multimodal browser, in the one or more markup documents the web resources for display in the multimodal browser further comprises identifying, for at least one of the web resources, a file type consistent with a web resource.

17. The computer program product of claim 13 wherein computer program instructions to perform sending, by the multimodal browser, the web resources to the identified target for the web resources using the identified mode of resource sharing further comprises sending, for each of the web resources, a reference to the web resource.