Sequence dependent data message consolidation in a voice activated computer network environment

• US 10,630,751 B2
• Filed: 12/30/2016
• Issued: 04/21/2020
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bhaya, Gaurav, Stets, Robert
• Primary Examiner(s): Madamba, Glenford J
• Assistant Examiner(s): Chouat, Abderrahmen
• Application Number:
  US15/395,725
• Publication Number:
  US 20180191804A1
• Time in Patent Office: 1,208 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context

Abstract

Systems and methods to combine multiple voice activated audio input data packets that indicate sequence dependent operations are provided. A natural language processor component can receive first and second input audio signal from a client computing device, and can identify respective requests and corresponding trigger keywords. A direct action application programming interface (“API”) can generate respective action data structures, and can construct respective data transmissions including the respective action data structures. A thread optimization component can obtain data packets of the first data transmission, and can obtain data packets of the second data transmission. The thread optimization component can determine, based on a heuristic technique applied to the data packets of the respective data transmissions a sequence dependency parameter. The thread optimization component can merge, based on a comparison of the sequence dependency parameter with a threshold, the first and second data transmissions into a single thread.

Claims

1. A system to combine multiple voice activated audio input data packets that indicate sequence dependent operations, comprising:

a natural language processor component executed by a data processing system to receive, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a client computing device;

the natural language processor component to parse the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request;

a direct action application programming interface (“

API”

) of the data processing system to;

generate, based on the first trigger keyword and in response to the first request, a first action data structure; and

construct a first data transmission including the first action data structure to establish a first communication session between a first service provider computing device and the client computing device, data packets of the first communication session routed from the client computing device to the first service provider computing device via the data processing system;

the natural language processor component to receive, via the interface of the data processing system, data packets comprising a second input audio signal detected by the sensor of the client computing device, and to parse the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request;

the direct action API to;

generate, based on the second trigger keyword and in response to the second request, a second action data structure; and

construct a second data transmission including the second action data structure to establish a second communication session between a second service provider computing device and the client computing device, data packets of the second communication session routed from the client computing device to the second service provider computing device via the data processing system;

a thread optimization component executed by the data processing system to;

obtain data packets of the first data transmission including the first action data structure corresponding to the first trigger keyword identified by the natural language processor component, and to obtain data packets of the second data transmission including the second action data structure including the second trigger keyword identified by the natural language processor component;

determine, based on a heuristic technique applied to the first action data structure of the first data transmission and the second action data structure of the second data transmission, a sequence dependency parameter indicating that an action associated with the first action data structure generated by the direct action API is scheduled to occur prior to or after an action associated with the second action data structure generated by the direct action API; and

merge, based on a comparison of the sequence dependency parameter with a threshold, the first data transmission with the second data transmission into a single thread.

2. The system of claim 1, wherein the single thread indicates a first action scheduled to occur prior to a second action, the first action corresponding to the first action data structure and the second action corresponding to the second action data structure, comprising:

the direct action API to detect a completion of the first action; and

the interface of the data processing system to transmit the data packets of the single thread to the second service provider computing device subsequent to the completion of the first action.

3. The system of claim 1, wherein the sequence dependency parameter indicates that the action corresponding to the first action data structure is scheduled to occur prior to the action corresponding to the second action data structure.

4. The system of claim 3, comprising:

the direct action API to identify a condition associated with the action corresponding to the second action data structure based on a time associated with the action corresponding to the first action data structure, the condition indicating that the action of the second action data structure cannot proceed until the action of the first action data structure is complete.

5. The system of claim 4, comprising:

the interface of the data processing system to prevent transmission of the data packets of the second communication session to the second service provider computing device responsive to the condition.

6. The system of claim 4, comprising:

the direct action API to obtain an indication of termination of the condition; and

the interface of the data processing system to transmit the data packets of the second communication session to the second service provider computing device subsequent to termination of the condition.

7. The system of claim 1, wherein the single thread indicates sequence dependent actions for the single thread in a sequential order, the sequential order including a first action, a second action subsequent to the first action, and a third action subsequent to the second action, comprising the direct action API to:

receive an indication of the sequential order, and subsequent to receipt of the indication of sequential order, generate a modified sequential order, the modified sequential order including the third action prior to the second action.

8. The system of claim 1, wherein the single thread indicates a series of sequence dependent actions for the single thread, the series of sequence dependent actions including a first action, a second action subsequent to the first action, and a third action subsequent to the second action, comprising:

the direct action API to determine a condition based on a time of completion of the first action, the condition indicating that the second action cannot proceed until the first action is complete;

the direct action API to determine a release of the condition based on completion of the first action; and

a content selector component of the data processing system to identify a content item for the second action, responsive to the release of the condition.

9. The system of claim 1, comprising:

the thread optimization component to identify a series of sequence dependent actions for the single thread.

10. The system of claim 1, comprising:

the direct action API to transmit at least one of the first data transmission and the second data transmission as part of the first communication session subsequent to merger of the first data transmission and the second data transmission into the single thread.

11. The system of claim 1, comprising:

the direct action API to identify a condition related to a sequence of a plurality of actions, the condition indicating that a second action of the sequence of the plurality of actions cannot proceed until a first action of the sequence of the plurality of actions is complete; and

a content selector component of the data processing system to block selection of a content item associated with series of sequence dependent actions for the single thread responsive to the condition.

12. The system of claim 1, comprising:

the thread optimization component to prioritize the second action data structure over the first action data structure.

13. The system of claim 12, comprising a content selector component of the data processing system to:

identify a content item responsive to the second action data structure; and

subsequent to identification of the content item responsive to the second action data structure, identify a content item responsive to the first action data structure.

14. The system of claim 1, comprising:

a content selector component of the data processing system to select a content item responsive to the second action data structure prior to occurrence of the action of the second action data structure.

15. The system of claim 1, comprising:

the direct action API to populate a variable field of at least one of the first action data structure and the second action data structure based on a sequence dependency between a first action and a second action.

16. A method of managing multiple voice activated audio input data communications that indicate sequence dependent operations, comprising:

receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising a first input audio signal detected by a sensor of a client computing device;

identifying, by the natural language processor component, from the first input audio signal, a first request and a first trigger keyword corresponding to the first request;

generating, by a direct action application programming interface (“

API”

) of the data processing system, based on the first trigger keyword and in response to the first request, a first action data structure;

constructing a first data transmission including the first action data structure for a first communication session between a first service provider computing device and the client computing device, data packets of the first communication session routed from the client computing device to the first service provider computing device via the data processing system;

receiving, by the natural language processor component, via the interface of the data processing system, data packets comprising a second input audio signal detected by the sensor of the client computing device, and parsing, by the natural language processor component, the second input audio signal to identify a second request and a second trigger keyword corresponding to the second request;

generating, by the direct action API, based on the second trigger keyword and in response to the second request, a second action data structure;

constructing a second data transmission including the second action data structure for a second communication session between a second service provider computing device and the client computing device, data packets of the second communication session routed from the client computing device to the second service provider computing device via the data processing system;

obtaining, by a thread optimization component executed by the data processing system, data of the first data transmission including the first action data structure corresponding to the first trigger keyword identified by the natural language processor component, and data of the second data transmission including the second action data structure including the second trigger keyword identified by the natural language processor component;

determining, based on a heuristic technique applied to the first action data structure of the first data transmission and applied to the second action data structure of the second data transmission, a sequence dependency parameter indicating that an action associated with the first action data structure generated by the direct action API is scheduled to occur prior to or after an action associated with the second action data structure generated by the direct action API; and

associating, based on the sequence dependency parameter, the first action data structure and the second action data structure with a single thread.

17. The method of claim 16, wherein the single thread indicates a first action scheduled to occur prior to a second action, the first action corresponding to the first action data structure and the second action corresponding to the second action data structure, comprising:

detecting a completion of the first action; and

providing the data packets of the second communication session to the second service provider computing device subsequent to the completion of the first action.

18. The method of claim 16, comprising:

identifying a condition associated with an action of the single thread, the condition indicating that the action cannot proceed until a prior action of the single thread is complete.

19. The method of claim 16, wherein the single thread indicates sequence dependent actions for the single thread in a sequential order, the sequential order including a first action, a second action subsequent to the first action, and a third action subsequent to the second action, comprising:

receiving an indication of the sequential order; and

generating a modified sequential order, the modified sequential order including the third action prior to the second action.