Voice-enabled dialog interaction with web pages

• US 9,690,854 B2
• Filed: 11/27/2013
• Issued: 06/27/2017
• Est. Priority Date: 11/27/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Melamed, I. Dan, Jung, Hyuckchul, Niraula, Nobal Bikram, Stent, Amanda Joy
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US14/092,033
• Publication Number:
  US 20150149168A1
• Time in Patent Office: 1,308 Days
• Field of Search: 704235, 704 9, 704276, 704275, 719328, 715773, 715234, 345649, 345536
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 16/9574 of access to content, e.g. by caching
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Voice enabled dialog with web pages is provided. An Internet address of a web page is received including an area with which a user of a client device can specify information. The web page is loaded using the received Internet address of the web page. A task structure of the web page is then extracted. An abstract representation of the web is then generated. A dialog script, based on the abstract representation of the web page is then provided. Spoken information received from the user is converted into text and the converted text is inserted into the area.

Claims

1. A method of enabling voice interaction with a web page, comprising:

receiving, at a processor, an Internet address of the web page, the web page including an area with which a user of a client device can specify information;

determining that the client device does not have an alphabetic input device and a displayable keypad;

loading the web page using the Internet address;

extracting a task structure of the web page;

generating a representation of the web page to facilitate the voice interaction with the web page; and

providing a dialog script based on the representation to provide the voice interaction with the user,

wherein spoken information of the user is converted into text and the text is inserted into the area.

2. The method of claim 1, further comprising:

extracting forms, form constraints, and dependencies from the web page using a browser object model and a document object model.

3. The method of claim 1, further comprising:

generating a browser object model of the web page.

4. The method of claim 3, wherein the browser object model comprises a state of hypertext markup language elements as the hypertext markup language elements would appear in a web browser.

5. The method of claim 3, further comprising:

generating a document object model of the web page.

6. The method of claim 5, wherein the document object model comprises a tree of hypertext markup language elements.

7. The method of claim 1, further comprising:

detecting that the user of the client device prefers to specify information in the web page by voice.

8. The method of claim 1, further comprising:

determining that the client device is a portable communications device.

9. The method of claim 8, wherein the determining that the client device is a portable communications device is based upon a received user agent string in a hypertext transfer protocol address header received with the Internet address.

10. A system for enabling voice interaction with a web page, comprising:

a processor;

a memory that stores executable instructions that, when executed by the processor, cause the processor to perform operations comprising;

receiving an Internet address of the web page, the web page including an area with which a user of a client device can specify information;

determining that the client device does not have an alphabetic input device and a displayable keypad;

loading the web page using the Internet address;

extracting a task structure of the web page;

generating a representation of the web page to facilitate the voice interaction with the web page; and

providing a dialog script based on the representation to provide the voice interaction with the user, and

wherein spoken information of the user is converted into text and the text is inserted into the area.

11. A non-transitory computer readable storage medium encoded with an executable computer program that enables voice interaction with a web page and that, when executed by a processor, causes the processor to perform operations comprising:

receiving an Internet address of the web page, the web page including an area with which a user of a client device can specify information;

determining that the client device does not have an alphabetic input device and a displayable keypad;

loading the web page using the Internet address;

extracting a task structure of the web page;

generating a representation of the web page to facilitate the voice interaction with the web page; and

providing a dialog script based on the representation to provide the voice interaction with the user,

wherein spoken information of the user is converted into text and the text is inserted into the area.

12. The non-transitory computer readable storage medium of claim 11, wherein the operations further comprise:

extracting forms, form constraints, and dependencies from the web page using a browser object model and a document object model.

13. The non-transitory computer readable storage medium of claim 11, wherein the operations further comprise:

generating a browser object model of the web page.

14. The non-transitory computer readable storage medium of claim 13, wherein the browser object model comprises a state of hypertext markup language elements as the hypertext markup language elements would appear in a web browser.

15. The non-transitory computer readable storage medium of claim 13, wherein the operations further comprise:

generating a document object model of the web page.

16. The non-transitory computer readable storage medium of claim 11, wherein the operations further comprise:

detecting that the user of the client device prefers to specify information in the web page by voice.

17. The non-transitory computer readable storage medium of claim 11, wherein the operations further comprise:

determining that the client device is a portable communications device.

18. The non-transitory computer readable storage medium of claim 17, wherein the determining that the client device is a portable communications device is based upon a received user agent string in a hypertext transfer protocol address header received with the Internet address.