Graphical user interface rendering management by voice-driven computing infrastructure

• US 10,558,426 B2
• Filed: 12/08/2017
• Issued: 02/11/2020
• Est. Priority Date: 12/08/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kothari, Anshul, Bhaya, Gaurav, Jain, Tarun
• Primary Examiner(s): Silverman, Seth A
• Application Number:
  US15/836,746
• Publication Number:
  US 20190179608A1
• Time in Patent Office: 795 Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 67/53 using third party service providers

Abstract

Managing rendering of a graphical user interface is provided. A system receives data packets comprising an input audio signal. The system determines an application identifier and query. The system provides the query to the application to cause the application to generate a second query for transmission to a third-party server, and identify responses to the query. The system intercepts the responses, and generates a keyword based on the responses. The system selects a digital component using the keyword, executes a deduplication process, and determines to add the digital component to the responses. The system constructs a display output using a graphical user interface template that integrates the plurality of responses generated by the application with the digital component, and provides the display output to the computing device for rendering.

Claims

1. A system to manage rendering of a graphical user interface in a voice-driven computing environment, comprising:

a data processing system comprising one or more processors and memory to execute a natural language processor component, an interface, an output merger component, and a content selector component, the data processing system to;

receive, via the interface, data packets comprising an input audio signal detected by a sensor of a computing device;

determine, from the input audio signal, an application identifier and a first query;

generate a second query based on the first query and information associated with the computing device;

provide the second query to a third-party server that provides an application corresponding to the application identifier to cause the application to identify a plurality of responses to the second query for display via the computing device;

obtain, prior to display on the computing device and from the application, the plurality of responses identified by the application responsive to the second query transmitted to the third-party server;

parse the plurality of responses to generate a keyword based on the plurality of responses;

execute, responsive to generation of the keyword, a real-time content selection process with the keyword generated based on the plurality of responses obtained from the third-party server;

identify, based on the real-time content selection process, a digital component provided by a content provider device;

execute a deduplication process with the digital component and the plurality of responses generated by the application;

determine, responsive to the deduplication process executed with the digital component and the plurality of responses generated by the application, to add the digital component to the plurality of responses;

retrieve, for the application, a graphical user interface template defining a font, a color, and a layout to render the plurality of responses;

construct a display output using the graphical user interface template that integrates the plurality of responses generated by the application with the digital component identified based on the real-time content selection process; and

provide the display output to the computing device to cause the computing device to render the display output for display via a display device communicatively coupled to the computing device.

2. The system of claim 1, comprising the data processing system to:

parse the first query to identify a location-related term;

determine, based on identification of the location-related term, a location of the computing device; and

insert the location of the computing device into the first query to generate the second query.

3. The system of claim 1, comprising the data processing system to:

transmit, to the third-party server that provides the application, a request for the graphical user interface template;

receive, responsive to the request, the graphical user interface template; and

store the graphical user interface template in a data repository of the data processing system.

4. The system of claim 1, comprising the data processing system to:

modify at least one of the font and the color of the digital component to integrate the digital component with the plurality of responses for the display output.

5. The system of claim 1, wherein the application comprises a third-party chatbot provided by the third-party server.

6. The system of claim 1, comprising the data processing system to:

select a position among the plurality of responses; and

add the digital component to the position among the plurality of responses to construct the display output.

7. The system of claim 1, comprising the data processing system to:

select, based on a machine learning model generated with historical performance data collected from a plurality of instances of the application, a position among the plurality of responses; and

add the digital component to the position among the plurality of responses to construct the display output.

8. The system of claim 1, comprising:

the data processing system to parse each of the plurality of responses using a semantic analysis technique to identify the keyword.

9. The system of claim 1, comprising the data processing system to:

generate a tuple formed from the application identifier and an identifier of the computing device; and

select the graphical user interface template based on the tuple formed from the application identifier and an identifier of the computing device.

10. The system of claim 1, comprising the data processing system to:

receive a second digital component responsive to a second plurality of responses selected based on a second input signal detected by the sensor of the computing device;

execute a second deduplication process with the second digital component and the second plurality of responses generated by the application;

determine, responsive to the second deduplication process executed with the digital component and the plurality of responses generated by the application, that the second digital component matches with one of the second plurality of responses; and

block addition of the second digital component to the second plurality of responses.

11. The system of claim 1, comprising the data processing system to:

receive a second digital component responsive to a second plurality of responses selected based on a second input signal detected by the sensor of the computing device;

execute a second deduplication process with the second digital component and the second plurality of responses generated by the application;

determine, responsive to the second deduplication process executed with the digital component and the plurality of responses generated by the application, that the second digital component matches with one of the second plurality of responses;

select a third digital component responsive to the determination that the second digital component matches with one of the second plurality of responses; and

add the third digital component to the second plurality of responses for display via the display device.

12. A method of managing rendering of a graphical user interface in a voice-driven computing environment, comprising:

receiving, via an interface of a data processing system comprising one or more processors, data packets comprising an input audio signal detected by a sensor of a computing device;

determine, by the data processing system from the input audio signal, an application identifier and a first query for input into an application corresponding to the application identifier;

generating, by the data processing system, a second query based on the first query;

providing, by the data processing system, the second query to a third-party server to cause the third-party server to identify a plurality of responses to the second query for display via the computing device;

obtaining, by the data processing system, prior to display on the computing device and from the application, the plurality of responses identified by the application responsive to the second query transmitted to the third-party server;

parsing, by the data processing system, the plurality of responses to generate a keyword based on the plurality of responses;

executing, by the data processing system, responsive to generation of the keyword, a real-time content selection process with the keyword generated based on the plurality of responses intercepted from the application;

identifying, by the data processing system, based on the real-time content selection process, a digital component provided by a content provider device;

executing, by the data processing system, a deduplication process with the digital component and the plurality of responses generated by the application;

determining, by the data processing system, responsive to the deduplication process executed with the digital component and the plurality of responses generated by the application, to add the digital component to the plurality of responses;

retrieving, by the data processing system, for the application, a graphical user interface template defining a font, color, and a layout to render the plurality of responses;

constructing, by the data processing system, a display output using the graphical user interface template that integrates the plurality of responses generated by the application with the digital component identified based on the real-time content selection process; and

providing, by the data processing system, the display output to the computing device to cause the computing device to render the display output for display via a display device communicatively coupled to the computing device.

13. The method of claim 12, comprising:

modifying at least one of the font and the color of the digital component to integrate the digital component with the plurality of responses for the display output.

14. The method of claim 12, wherein the application comprises a third-party chatbot provided by the third-party server.

15. The method of claim 12, comprising:

selecting a position among the plurality of responses; and

adding the digital component to the position among the plurality of responses to construct the display output.

16. The method of claim 12, comprising:

selecting, based on a machine learning model generated with historical performance data collected from a plurality of instances of the application, a position among the plurality of responses; and

adding the digital component to the position among the plurality of responses to construct the display output.

17. The method of claim 12, comprising:

parsing each of the plurality of responses using a semantic analysis technique to identify the keyword.

18. The method of claim 12, comprising:

generating a tuple formed from the application identifier and an identifier of the computing device; and

selecting the graphical user interface template based on the tuple formed from the application identifier and an identifier of the computing device.

19. A digital assistant device, comprising:

a display device;

a graphics driver;

a sensor to detect an input audio signal; and

a pre-processor component coupled to the display device, the graphics driver, and the sensor, the pre-processor component to;

filter the input audio signal to create a filtered input audio signal;

convert the filtered input audio signal to data packets;

transmit the data packets to a data processing system comprising one or more processors and memory that execute a natural language processor component, an interface, an output merger component, and a content selector component, the data processing system to;

receive, via the interface, data packets comprising the input audio signal detected by the sensor of the computing device;

determine, from the input audio signal, an application identifier and a first query;

generate a second query based on the first query and information associated with the computing device;

provide the second query to a third-party server that provides an application corresponding to the application identifier to cause the application to identify a plurality of responses to the second query for display via the computing device;

obtain, prior to display on the computing device and from the application, the plurality of responses identified by the application responsive to the second query transmitted to the third-party server;

parse the plurality of responses to generate a keyword based on the plurality of responses;

execute, responsive to generation of the keyword, a real-time content selection process with the keyword generated based on the plurality of responses intercepted from the application;

identify, based on the real-time content selection process, a digital component provided by a content provider device;

execute a deduplication process with the digital component and the plurality of responses generated by the application;

determine, responsive to the deduplication process executed with the digital component and the plurality of responses generated by the application, to add the digital component to the plurality of responses;

retrieve, for the application, a graphical user interface template defining a font, color, and layout to render the plurality of responses;

construct a display output using the graphical user interface template that integrates the plurality of responses generated by the application with the digital component identified based on the real-time content selection process; and

provide the display output to the digital assistant device to cause the digital assistant device to render the display output for display via the display device communicatively coupled to the digital assistant device; and

the graphics driver to render the display output to cause the display device to present a visual indication corresponding to the display output.

20. The device of claim 19, comprising:

an audio driver;

a speaker component; and

the pre-processor component to receive an indication of the display output and instruct the audio driver to generate an output audio signal to cause the speaker component to transmit an audio output corresponding to the indication of the display output.