Presenting Supplemental Content For Digital Media Using A Multimodal Application

• US 20080208589A1
• Filed: 02/27/2007
• Published: 08/28/2008
• Est. Priority Date: 02/27/2007
• Status: Abandoned Application

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Jania, Frank L., Goodman, Brian D., Cross, Charles W., Shaw, Darren M.
• Application Number:
  US11/679,225
• Publication Number:
  US 20080208589A1
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/4126 The peripheral being portable, e.g. PDAs or mobile phones
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 21/8543 using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80
• CPC Class Codes: H04N 5/45 Picture in picture , e.g. displaying simultaneously another television channel in a region of the screen
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Presenting supplemental content for digital media using a multimodal application, implemented with a grammar of the multimodal application in an automatic speech recognition (‘ASR’) engine, with the multimodal application operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to the ASR engine, includes: rendering, by the multimodal application, a portion of the digital media; receiving, by the multimodal application, a voice utterance from a user; determining, by the multimodal application using the ASR engine, a recognition result in dependence upon the voice utterance and the grammar; identifying, by the multimodal application, supplemental content for the rendered portion of the digital media in dependence upon the recognition result; and rendering, by the multimodal application, the supplemental content.

Claims

1. A method of presenting supplemental content for digital media using a multimodal application, the method implemented with a grammar of the multimodal application in an automatic speech recognition (‘

ASR’

) engine, with the multimodal application operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to the ASR engine, the method comprising;

rendering, by the multimodal application, a portion of the digital media;

receiving, by the multimodal application, a voice utterance from a user;

determining, by the multimodal application using the ASR engine, a recognition result in dependence upon the voice utterance and the grammar;

identifying, by the multimodal application, supplemental content for the rendered portion of the digital media in dependence upon the recognition result; and

rendering, by the multimodal application, the supplemental content.

2. The method of claim 1 wherein rendering, by the multimodal application, the supplemental content further comprises supplementing the rendered portion of digital media with the supplemental content.

3. The method of claim 1 wherein the supplemental content further comprises annotated content for the digital media.

4. The method of claim 1 wherein the supplemental content further comprises another portion of the digital media.

5. The method of claim 1 wherein identifying, by the multimodal application, supplemental content for the digital media in dependence upon the recognition result further comprises querying a content repository for supplemental content associated with at least a portion of the recognition result.

6. The method of claim 1 wherein identifying, by the multimodal application, supplemental content for the digital media in dependence upon the recognition result further comprises searching the digital media for supplemental content associated with at least a portion of the recognition result.

7. The method of claim 1 wherein the grammar further comprises grammar rules, the grammar rules specifying recognition results according to the supplemental content.

8. The method of claim 1 wherein the digital media is digital video.

9. Apparatus for presenting supplemental content for digital media using a multimodal application, the method implemented with a grammar of the multimodal application in an automatic speech recognition (‘

ASR’

) engine, with the multimodal application operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to the ASR engine, the apparatus comprising a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions capable of;

rendering, by the multimodal application, a portion of the digital media;

receiving, by the multimodal application, a voice utterance from a user;

determining, by the multimodal application using the ASR engine, a recognition result in dependence upon the voice utterance and the grammar;

identifying, by the multimodal application, supplemental content for the rendered portion of the digital media in dependence upon the recognition result; and

rendering, by the multimodal application, the supplemental content.

10. The apparatus of claim 9 wherein rendering, by the multimodal application, the supplemental content further comprises supplementing the rendered portion of digital media with the supplemental content.

11. The apparatus of claim 9 wherein identifying, by the multimodal application, supplemental content for the digital media in dependence upon the recognition result further comprises querying a content repository for supplemental content associated with at least a portion of the recognition result.

12. The apparatus of claim 9 wherein identifying, by the multimodal application, supplemental content for the digital media in dependence upon the recognition result further comprises searching the digital media for supplemental content associated with at least a portion of the recognition result.

13. The apparatus of claim 9 wherein the grammar further comprises grammar rules, the grammar rules specifying recognition results according to the supplemental content.

14. The apparatus of claim 9 wherein the digital media is digital video.

15. A computer program product for presenting supplemental content for digital media using a multimodal application, the method implemented with a grammar of the multimodal application in an automatic speech recognition (‘

ASR’

) engine, with the multimodal application operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal application operatively coupled to the ASR engine, the computer program product disposed upon a recordable medium, the computer program product comprising computer program instructions capable of;

rendering, by the multimodal application, a portion of the digital media;

receiving, by the multimodal application, a voice utterance from a user;

determining, by the multimodal application using the ASR engine, a recognition result in dependence upon the voice utterance and the grammar;

identifying, by the multimodal application, supplemental content for the rendered portion of the digital media in dependence upon the recognition result; and

rendering, by the multimodal application, the supplemental content.

16. The computer program product of claim 15 wherein rendering, by the multimodal application, the supplemental content further comprises supplementing the rendered portion of digital media with the supplemental content.

17. The computer program product of claim 15 wherein identifying, by the multimodal application, supplemental content for the digital media in dependence upon the recognition result further comprises querying a content repository for supplemental content associated with at least a portion of the recognition result.

18. The computer program product of claim 15 wherein identifying, by the multimodal application, supplemental content for the digital media in dependence upon the recognition result further comprises searching the digital media for supplemental content associated with at least a portion of the recognition result.

19. The computer program product of claim 15 wherein the grammar further comprises grammar rules, the grammar rules specifying recognition results according to the supplemental content.

20. The computer program product of claim 15 wherein the digital media is digital video.