Speech-Enabled Web Content Searching Using A Multimodal Browser

• US 20080228494A1
• Filed: 03/13/2007
• Published: 09/18/2008
• Est. Priority Date: 03/13/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Cross, Charles W.
• Granted Patent: US 8,843,376 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Speech-enabled web content searching using a multimodal browser implemented with one or more grammars in an automatic speech recognition (‘ASR’) engine, with the multimodal browser operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal browser operatively coupled to the ASR engine, includes: rendering, by the multimodal browser, web content; searching, by the multimodal browser, the web content for a search phrase, including yielding a matched search result, the search phrase specified by a first voice utterance received from a user and a search grammar; and performing, by the multimodal browser, an action in dependence upon the matched search result, the action specified by a second voice utterance received from the user and an action grammar.

Claims

1. A method of speech-enabled web content searching using a multimodal browser, the method implemented with one or more grammars in an automatic speech recognition (‘

ASR’

) engine, with the multimodal browser operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal browser operatively coupled to the ASR engine, the method comprising;

rendering, by the multimodal browser, web content;

searching, by the multimodal browser, the web content for a search phrase, including yielding a matched search result, the search phrase specified by a first voice utterance received from a user and a search grammar; and

performing, by the multimodal browser, an action in dependence upon the matched search result, the action specified by a second voice utterance received from the user and an action grammar.

2. The method of claim 1 wherein searching, by the multimodal browser, the web content for a search phrase, including yielding a matched search result further comprises:

creating the search grammar in dependence upon the web content;

receiving the first voice utterance from a user;

determining, using the ASR engine, the search phrase in dependence upon the first voice utterance and the search grammar; and

matching the search phrase to at least one portion of the web content, yielding a matched search result.

3. The method of claim 2 wherein matching the search phrase to at least one portion of the web content, yielding a matched search result further comprises identifying a node of a Document Object Model (‘

DOM’

) representing the web content that contains the search phrase.

4. The method of claim 1 wherein performing, by the multimodal browser, an action in dependence upon the matched search result further comprises:

creating the action grammar in dependence upon the matched search result;

receiving the second voice utterance from the user;

determining, using the ASR engine, an action identifier in dependence upon the second voice utterance and the action grammar; and

performing the specified action in dependence upon the action identifier.

5. The method of claim 1 further comprising augmenting, by the multimodal browser, the matched search result with additional web content.

6. The method of claim 5 wherein augmenting, by the multimodal browser, the matched search result with additional web content further comprises inserting the additional web content into a node of a Document Object Model (‘

DOM’

) representing the web content that contains the matched search result.

7. The method of claim 1 wherein the web content is not speech-enabled.

8. Apparatus for speech-enabled web content searching using a multimodal browser, the method implemented with one or more grammars in an automatic speech recognition (‘

ASR’

) engine, with the multimodal browser operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal browser operatively coupled to the ASR engine, the apparatus comprising a computer processor and a computer memory operatively coupled to the computer processor, the computer memory having disposed within it computer program instructions capable of;

rendering, by the multimodal browser, web content;

searching, by the multimodal browser, the web content for a search phrase, including yielding a matched search result, the search phrase specified by a first voice utterance received from a user and a search grammar; and

performing, by the multimodal browser, an action in dependence upon the matched search result, the action specified by a second voice utterance received from the user and an action grammar.

9. The apparatus of claim 8 wherein searching, by the multimodal browser, the web content for a search phrase, including yielding a matched search result further comprises:

creating the search grammar in dependence upon the web content;

receiving the first voice utterance from a user;

determining, using the ASR engine, the search phrase in dependence upon the first voice utterance and the search grammar; and

matching the search phrase to at least one portion of the web content, yielding a matched search result.

10. The apparatus of claim 9 wherein matching the search phrase to at least one portion of the web content, yielding a matched search result further comprises identifying a node of a Document Object Model (‘

DOM’

) representing the web content that contains the search phrase.

11. The apparatus of claim 8 wherein performing, by the multimodal browser, an action in dependence upon the matched search result further comprises:

creating the action grammar in dependence upon the matched search result;

receiving the second voice utterance from the user;

determining, using the ASR engine, an action identifier in dependence upon the second voice utterance and the action grammar; and

performing the specified action in dependence upon the action identifier.

12. The apparatus of claim 8 further comprising computer program instructions capable of augmenting, by the multimodal browser, the matched search result with additional web content.

13. The apparatus of claim 12 wherein augmenting, by the multimodal browser, the matched search result with additional web content further comprises inserting the additional web content into a node of a Document Object Model (‘

DOM’

) representing the web content that contains the matched search result

14. A computer program product for speech-enabled web content searching using a multimodal browser, the method implemented with one or more grammars in an automatic speech recognition (‘

ASR’

) engine, with the multimodal browser operating on a multimodal device supporting multiple modes of interaction including a voice mode and one or more non-voice modes, the multimodal browser operatively coupled to the ASR engine, the computer program product disposed upon a computer-readable medium, the computer program product comprising computer program instructions capable of;

rendering, by the multimodal browser, web content;

searching, by the multimodal browser, the web content for a search phrase, including yielding a matched search result, the search phrase specified by a first voice utterance received from a user and a search grammar; and

performing, by the multimodal browser, an action in dependence upon the matched search result, the action specified by a second voice utterance received from the user and an action grammar.

15. The computer program product of claim 14 wherein searching, by the multimodal browser, the web content for a search phrase, including yielding a matched search result further comprises:

creating the search grammar in dependence upon the web content;

receiving the first voice utterance from a user;

determining, using the ASR engine, the search phrase in dependence upon the first voice utterance and the search grammar; and

matching the search phrase to at least one portion of the web content, yielding a matched search result.

16. The computer program product of claim 15 wherein matching the search phrase to at least one portion of the web content, yielding a matched search result further comprises identifying a node of a Document Object Model (‘

DOM’

) representing the web content that contains the search phrase.

17. The computer program product of claim 14 wherein performing, by the multimodal browser, an action in dependence upon the matched search result further comprises:

creating the action grammar in dependence upon the matched search result;

receiving the second voice utterance from the user;

determining, using the ASR engine, an action identifier in dependence upon the second voice utterance and the action grammar; and

performing the specified action in dependence upon the action identifier.

18. The computer program product of claim 14 further comprising computer program instructions capable of augmenting, by the multimodal browser, the matched search result with additional web content.

19. The computer program product of claim 18 wherein augmenting, by the multimodal browser, the matched search result with additional web content further comprises inserting the additional web content into a node of a Document Object Model (‘

DOM’

) representing the web content that contains the matched search result.

20. The computer program product of claim 14 wherein the web content is not speech-enabled.