Component libraries for voice interaction services

• US 10,261,752 B2
• Filed: 08/02/2016
• Issued: 04/16/2019
• Est. Priority Date: 08/02/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sung, Sang Soo, Zheng, Lantian, Chan, Haywai Hayward, Liu, Chen, Sun, Liuyi, Whipp, David P.
• Primary Examiner(s): Kujundzic, Dino
• Application Number:
  US15/226,046
• Publication Number:
  US 20180039477A1
• Time in Patent Office: 987 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech

Abstract

The disclosed embodiments include computerized methods, systems, and devices, including computer programs encoded on a computer storage medium, for integrating voice-based interaction and control into a native graphical user interface (GUI) of an executed application. For example, a communications device may obtaining component data identifying a plurality of components of a voice-user interface from a computing system maintained by a voice-service provider, and may execute an application linked to a corresponding one of the components of the voice-user interface. The communications device may generate the native GUI based on an output of the executed application, and may generate an interface element representative of the corresponding one of the components of the voice-user interface. The communications device may present the generated interface element within the native GUI, which may embed the corresponding component of the voice-user interface into the native GUI.

Claims

1. A computer-implemented method, comprising:

obtaining, by one or more processors, a library of components for a voice-user interface;

executing an application using the one or more processors,

wherein the application is linked to a particular one of the components from the library of components for the voice-user interface,

wherein executing the application includes accessing the library of components for the voice-user interface to retrieve code for the particular one of the components from the library of components for the voice-user interface that is linked to the application;

generating, by the one or more processors, a native graphical user interface for the executed application, the native graphical user interface comprising first interface elements, the first interface elements comprising content associated with the executed application;

generating, by the one or more processors, as a result of the particular one of the components from the library of components for the voice-user interface being linked to the application, a second interface element representative of the particular one of the components;

embedding, by the one or more processors, the second interface element in the native graphical user interface for the executed application; and

presenting, by the one or more processors, the native graphical user interface for the executed application through a display unit of a communications device, including presenting the second interface element representative of the particular one of the components from the library of components for the voice-user interface that is linked to the application along with the first interface elements within the native graphical user interface.

2. The method of claim 1, wherein:

generating the second interface element comprises generating layout data specifying a position of the particular one of the components from the library of components for the voice-user interface within the native graphical user interface; and

presenting the native graphical user interface comprises presenting the second interface element within the native graphical user interface at the specified position according to the layout data.

3. The method of claim 1, wherein the obtaining comprises receiving a least a portion of the library of components for the voice-user interface from a computing system associated with a voice-service provider.

4. The method of claim 3, wherein the receiving comprises receiving the portion of the library of components for the voice-user interface through a programmatic interface, the programmatic interface being established by the computing system associated with the voice-service provider.

5. The method of claim 1, wherein the application is dynamically linked to the particular one of the components from the library of components for the voice-user interface.

6. The method of claim 1, wherein:

the communications device comprises a microphone;

the second interface element comprises an icon associated with the microphone; and

the method further comprises;

detecting an operational status of the microphone; and

modifying a visual characteristic of the icon to reflect the detected operational status, the modification being visually perceptible by a user.

7. The method of claim 6, wherein:

the method further comprises;

receiving user input indicative of a selection of the icon; and

performing operations that activate the microphone in response to the received user input; and

the modifying comprises modifying the visual characteristics of the icon to reflect the activation of the microphone.

8. The method of claim 1, further comprising:

receiving audio data corresponding to a first utterance spoken by a user into a microphone of the communications device, the first utterance being associated with a functionality of the executed application;

obtaining structured data representative of the received audio data, the structured data arranged to cause the executed application to perform one or more operations consistent with the associated functionality; and

providing the structured data to the executed application through a programmatic interface, the executed application performing the one or more operations in accordance with the structured data.

9. The method of claim 8, further comprising:

identifying linguistic elements that represent the first utterance based on an application of at least one speech recognition algorithm to the received audio data; and

presenting portions of the identified linguistic elements within the second interface element in the native graphical user interface.

10. The method of claim 8, wherein the presented second interface element comprises a textual representation of at least one of a command or a query associated with the executed application, the textual representation prompting the user to speak the utterance.

11. The method of claim 8, wherein:

the method further comprises receiving additional audio data corresponding to a second utterance spoken by the user into the microphone, the second utterance specifying a generic query;

the second interface element comprises at least one of a textual or graphical representation of a response to the generic query; and

the presenting comprises presenting, within the native graphical user interface, the at least one textual or graphical representation of the response within a view container associated with the second interface element.

12. The method of claim 11, wherein the second interface element comprises at least one of an overlay interface element, a slide-up interface element, a slide-down interface element, or a drawer interface element.

13. The method of claim 1, wherein a developer of the application has linked the application to the particular one of the components from the library of components for the voice-user interface.

14. A communications device, comprising:

at least one processor; and

a memory storing executable instructions that, when executed by the at least one processor, causes the at least one processor to perform operations comprising;

obtaining a library of components for a voice-user interface;

executing an application,

wherein the application is linked to a particular one of the components from the library of components for the voice-user interface,

wherein executing the application includes accessing the library of components for the voice-user interface to retrieve code for the particular one of the components from the library of components for the voice-user interface that is linked to the application;

generating a native graphical user interface for the executed application, the native graphical user interface comprising first interface elements, the first interface elements comprising content associated with the executed application;

generating, as a result of the particular one of the components from the library of components for the voice-user interface being linked to the application, a second interface element representative of the particular one of the components;

embedding the second interface element in the native graphical user interface for the executed application; and

presenting the native graphical user interface for the executed application through a display unit of a communications device, including presenting the second interface element representative of the particular one of the components from the library of components for the voice-user interface that is linked to the application along with the first interface elements within the native graphical user interface.

15. The communications device of claim 14, wherein:

generating the second interface element comprises generating layout data specifying a position of the particular one of the components from the library of components for the voice-user interface within the native graphical user interface; and

presenting the native graphical user interface comprises presenting the second interface element within the native graphical user interface at the specified position according to the layout data.

16. The communications device of claim 14, wherein the obtaining comprises receiving a least a portion of the library of components for the voice-user interface from a computing system associated with a voice-service provider.

17. The communications device of claim 16, wherein the receiving comprises receiving the portion of the library of components for the voice-user interface through a programmatic interface, the programmatic interface being established by the computing system associated with the voice-service provider.

18. The communications device of claim 14, wherein the application is dynamically linked to the particular one of the components from the library of components for the voice-user interface.

19. One or more non-transitory computer-readable media storing instructions that, when executed by at least one processor of a client device, cause performance of operations comprising:

obtaining a library of components for a voice-user interface;

executing an application,

wherein the application is linked to a particular one of the components from the library of components for the voice-user interface,

wherein executing the application includes accessing the library of components for the voice-user interface to retrieve code for the particular one of the components from the library of components for the voice-user interface that is linked to the application;

generating a native graphical user interface for the executed application, the native graphical user interface comprising first interface elements, the first interface elements comprising content associated with the executed application;

generating, as a result of the particular one of the components from the library of components for the voice-user interface being linked to the application, a second interface element representative of the particular one of the components;

embedding the second interface element in the native graphical user interface for the executed application; and

presenting the native graphical user interface for the executed application through a display unit of a communications device, including presenting the second interface element representative of the particular one of the components from the library of components for the voice-user interface that is linked to the application along with the first interface elements within the native graphical user interface.