Sequence dependent data message consolidation in a voice activated computer network environment

• US 10,893,088 B2
• Filed: 03/13/2020
• Issued: 01/12/2021
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bhaya, Gaurav, Stets, Jr., Robert James
• Primary Examiner(s): Parry, Chris
• Assistant Examiner(s): Chouat, Abderrahmen
• Application Number:
  US16/818,736
• Publication Number:
  US 20200213382A1
• Time in Patent Office: 305 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context

Abstract

Systems and methods to combine multiple voice activated audio input data packets that indicate sequence dependent operations are provided. A natural language processor component can receive first and second input audio signal from a client computing device, and can identify respective requests and corresponding trigger keywords. A direct action application programming interface (“API”) can generate respective action data structures, and can construct respective data transmissions including the respective action data structures. A thread optimization component can obtain data packets of the first data transmission, and can obtain data packets of the second data transmission. The thread optimization component can determine, based on a heuristic technique applied to the data packets of the respective data transmissions a sequence dependency parameter. The thread optimization component can merge, based on a comparison of the sequence dependency parameter with a threshold, the first and second data transmissions into a single thread.

Claims

1. A system to perform sequence dependent operations in a voice-based computing environment, comprising:

a natural language processor component executed by a data processing system to receive, via an interface of the data processing system, data packets comprising one or more input audio signals detected by a sensor of a client computing device;

the natural language processor component to parse the one or more input audio signals to identify one or more requests and one or more trigger keywords corresponding to the one or more requests;

a direct action application programming interface (“

API”

) of the data processing system to generate, based on the one or more trigger keywords and in response to the one or more requests, a first action data structure for a first action and a second action data structure for a second action; and

a thread optimization component executed by the data processing system to;

obtain data packets of the first action data structure, and to obtain data packets of the second action data structure;

determine, based on a heuristic technique applied to the data packets of the first action data structure and the data packets of the second action data structure, a sequence dependency parameter;

combine, based on a comparison of the sequence dependency parameter with a threshold, the data packets of the first action data structure and the data packets of the second action data structure into a single thread; and

identify a series of sequence dependent actions for the single thread;

a content selector component to select a content item for at least one of the series of sequence dependent actions; and

the data processing system to;

provide the content item to the client computing device;

identify a condition associated with the second action of the second action data structure based on a time associated with the first action of the first action data structure;

receive an indication of termination of the condition; and

transmit the data packets of the second action data structure to a service provider computing device subsequent to the termination of the condition.

2. The system of claim 1, comprising the data processing system to:

generate the first action data structure based on a first trigger keyword of the one or more trigger keywords; and

construct a first data transmission including the first action data structure to establish a first communication session between a first service provider computing device and the client computing device, wherein data packets of the first communication session routed from the client computing device to the first service provider computing device via the data processing system; and

including the first data transmission in the single thread.

3. The system of claim 1, comprising the data processing system to:

generate the first action data structure based on a first trigger keyword of the one or more trigger keywords, and the second action data structure based on a second trigger keyword of the one or more trigger keywords.

4. The system of claim 1, comprising the data processing system to:

construct a first data transmission including the first action data structure to establish a first communication session between a first service provider computing device and the client computing device;

construct a second data transmission including the second action data structure to establish a second communication session between a second service provider computing device and the client computing device; and

merge, based on the comparison of the sequence dependency parameter with the threshold, the first data transmission with the second data transmission into the single thread.

5. The system of claim 1, comprising the data processing system to:

detect a completion of the first action of the first action data structure; and

transmit the data packets of the single thread to the service provider computing device subsequent to the completion of the first action, wherein the single thread indicates that the first action is scheduled to occur prior to the second action of the second action data structure.

6. The system of claim 1, wherein the sequence dependency parameter indicates that the first action corresponding to the first action data structure is scheduled to occur prior to the second action corresponding to the second action data structure.

7. The system of claim 1, comprising the data processing system to:

to identify a second condition associated with the second action corresponding to the second action data structure based on the time associated with the first action corresponding to the first action data structure; and

prevent transmission of the data packets of the second action data structure to the service provider computing device responsive to the second condition.

8. The system of claim 1, wherein the single thread indicates a series of sequence dependent actions for the single thread, the series of sequence dependent actions including the first action, the second action subsequent to the first action, and a third action subsequent to the second action.

9. The system of claim 1, comprising:

the data processing system to identify a content item for the second action data structure prior to execution of the first action data structure.

10. A method of performing sequence dependent operations in a voice-based computing environment, comprising:

receiving, by a data processing system via an interface, data packets comprising one or more input audio signals detected by a sensor of a client computing device;

parsing, by the data processing system, the one or more input audio signals to identify one or more requests and one or more trigger keywords corresponding to the one or more requests;

generating, by the data processing system, based on the one or more trigger keywords and in response to the one or more requests, a first action data structure for a first action and a second action data structure for a second action;

obtaining, by the data processing system, data packets of the first action data structure, and to obtain data packets of the second action data structure;

determining, by the data processing system, based on a heuristic technique applied to the data packets of the first action data structure and the data packets of the second action data structure, a sequence dependency parameter;

combining, by the data processing system, based on a comparison of the sequence dependency parameter with a threshold, the data packets of the first action data structure and the data packets of the second action data structure into a single thread;

identifying, by the data processing system, a series of sequence dependent actions for the single thread;

selecting, by the data processing system, a content item for at least one of the series of sequence dependent actions;

providing, by the data processing system, the content item to the client computing device;

identifying, by the data processing system, a condition associated with the second action of the second action data structure based on a time associated with the first action of the first action data structure;

receiving, by the data processing system, an indication of termination of the condition; and

transmitting, by the data processing system, the data packets of the second action data structure to a service provider computing device subsequent to the termination of the condition.

11. The method of claim 10, comprising:

generating the first action data structure based on a first trigger keyword of the one or more trigger keywords;

constructing a first data transmission including the first action data structure to establish a first communication session between a first service provider computing device and the client computing device, wherein data packets of the first communication session routed from the client computing device to the first service provider computing device via the data processing system; and

providing the first data transmission in the single thread.

12. The method of claim 10, comprising:

generating the first action data structure based on a first trigger keyword of the one or more trigger keywords, and the second action data structure based on a second trigger keyword of the one or more trigger keywords.

13. The method of claim 10, comprising:

constructing a first data transmission including the first action data structure to establish a first communication session between a first service provider computing device and the client computing device;

constructing a second data transmission including the second action data structure to establish a second communication session between a second service provider computing device and the client computing device; and

combining, based on the comparison of the sequence dependency parameter with the threshold, the first data transmission with the second data transmission into the single thread.

14. The method of claim 10, comprising:

detecting a completion of the first action of the first action data structure; and

transmitting the data packets of the single thread to the service provider computing device subsequent to the completion of the first action, wherein the single thread indicates that the first action is scheduled to occur prior to the second action.

15. The method of claim 10, wherein the sequence dependency parameter indicates that the first action corresponding to the first action data structure is scheduled to occur prior to the second action corresponding to the second action data structure.

16. The method of claim 10, comprising:

identifying a second condition associated with the second action corresponding to the second action data structure based on the time associated with the first action corresponding to the first action data structure; and

preventing transmission of the data packets of the second action data structure to the service provider computing device responsive to the second condition.

17. The method of claim 10, wherein the single thread indicates a series of sequence dependent actions for the single thread, the series of sequence dependent actions including the first action, the second action subsequent to the first action, and a third action subsequent to the second action.

18. The method of claim 10, comprising:

identifying a content item for the second action data structure prior to execution of the first action data structure.