Sequence dependent operation processing of packet based data message transmissions

• US 10,013,978 B1
• Filed: 12/30/2016
• Issued: 07/03/2018
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bhaya, Gaurav, Stets, Robert
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US15/395,682
• Publication Number:
  US 20180190276A1
• Time in Patent Office: 550 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Optimization of sequence dependent operations in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. A prediction component can determine a thread based on the trigger keyword and the request that includes a first action, a second action subsequent to the first action, and a third action subsequent to the second action. A content selector component can select, based on the third action and the trigger keyword, a content item. An audio signal generator component can generate an output signal comprising the content item. An interface can transmit the output signal to cause a client computing device to drive a speaker to generate an acoustic wave corresponding to the output signal prior to occurrence of at least one of the first action and the second action.

Claims

1. A system to optimize processing of sequence dependent operations in a voice activated data packet based computer network environment, comprising:

a natural language processor component executed by a data processing system to receive data packets comprising an input audio signal detected by a sensor of a client computing device;

the natural language processor component to parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

a prediction component to determine a thread based on the trigger keyword and the request, the thread comprising a first action, a second action subsequent to the first action, and a third action subsequent to the second action;

the prediction component to provide, to a content selector component of the data processing system, an indication of the third action prior to occurrence of at least one of the first action and the second action;

the content selector component to select, based on the third action and the trigger keyword identified by the natural language processor, a content item via a real-time content selection process;

an audio signal generator component executed by the data processing system to generate an output signal comprising the content item; and

an interface of the data processing system to transmit data packets comprising the output signal generated by the audio signal generator component and cause an audio driver component executed by the client computing device to drive a speaker of the client computing device to generate an acoustic wave corresponding to the output signal prior to occurrence of at least one of the first action and the second action.

2. The system of claim 1, comprising:

the data processing system to prevent transmission of the data packets comprising the output signal to the client computing device until one of expiration of a determined time period, and receipt of an authorization indication from the client device.

3. The system of claim 1, wherein the interface of the data processing system receives a packet based data message from the client computing device, entered as audio input into the client computing device, to cancel the third action.

4. The system of claim 1, wherein the natural language processor component receives a packet based data message to cancel the third action within a time interval triggered by one of the first action and the second action.

5. The system of claim 1, comprising:

the interface of the data processing system to transmit the data packets comprising the output signal generated by the audio signal generator component and cause the audio driver component executed by the client computing device to drive the speaker of the client computing device to generate the acoustic wave corresponding to the output signal subsequent to occurrence of the first action and prior to occurrence of the second action.

6. The system of claim 1, comprising:

the data processing system to receive, from the client computing device a data message that includes a response to the content item prior to occurrence of at least one of the first action and the second action.

7. The system of claim 1, comprising:

the data processing system to initiate the third action based on one of a time of completion of the first action and a time of completion of the second action.

8. The system of claim 1, comprising:

a direct action application programming interface of the data processing system to initiate the third action subsequent to completion of at least one of the first action and the second action.

9. The system of claim 1, comprising:

a direct action application programming interface of the data processing system to initiate activity corresponding to the third action based on at least one of processor utilization of the data processing system and a bandwidth metric of a computer network.

10. The system of claim 1, comprising:

a direct action application programming interface of the data processing system to initiate activity corresponding to the third action and to subsequently initiate activity corresponding to the second action.

11. The system of claim 1, wherein the content item is a first content item and the real-time content selection process is a first real-time content selection process, comprising:

the content selector component to select, for the first action, a second content item via a second real-time content selection process and to select, for the second action, a third content item via a third real-time selection process.

12. The system of claim 11, comprising:

the content selector component to select the second content item and the third content item subsequent to selection of the first content item, wherein the first content item is associated with the third action, the second content item is associated with the first action, and the third content item is associated with the second action.

13. The system of claim 1, wherein the acoustic wave corresponds to the content item.

14. The system of claim 13, comprising:

the data processing system to obtain a response to the content item and to route a packet based data message to a natural language processor component of a service provider computing device prior to completion of at least one of the first action and the second action.

15. The system of claim 13, comprising:

the data processing system to obtain a response to the content item and to route a packet based data message to a natural language processor component of a service provider computing device subsequent to completion of at least one of the first action and the second action.

16. A method to invoke actions for sequence dependent operations in a voice activated data packet based computer network environment, comprising:

receiving, by a natural language processor component executed by a data processing system data packets comprising an input audio signal detected by a sensor of a client computing device;

identifying, by the natural language processor component, based on the input audio signal, a request and a trigger keyword corresponding to the request;

determining, by a prediction component, a thread based on the trigger keyword and the request, the thread comprising a first action, a second action subsequent to the first action, and a third action subsequent to the second action;

providing, by the prediction component, to a content selector component of the data processing system, an indication of the third action prior to occurrence of at least one of the first action and the second action;

selecting, by the content selector component, based on the third action and the trigger keyword identified by the natural language processor, a content item via a real-time content selection process;

obtaining, by an audio signal generator component executed by the data processing system, an output signal comprising the content item; and

transmitting, via an interface of the data processing system, data packets comprising the output signal obtained by the audio signal generator component to cause an audio driver component executed by the client computing device to drive a speaker of at least one of the client computing device and a second client computing device to generate an acoustic wave corresponding to the output signal prior to occurrence of at least one of the first action and the second action.

17. The method of claim 16, comprising:

preventing, by the data processing system, transmission of the data packets comprising the output signal to the client computing device until one of expiration of a determined time period, and receipt of an authorization indication from the client device.

18. The method of claim 16, comprising:

receiving, by the natural language processor component from the client computing device, a packet based data message to cancel the third action within a time interval triggered by one of the first action and the second action.

19. The method of claim 16, comprising:

receiving, by the data processing system, from the client computing device, a data message that includes a response to the content item prior to occurrence of at least one of the first action and the second action.

20. The method of claim 16, comprising:

initiating, by the data processing system, the third action based on one of a time of completion of the first action and a time of completion of the second action.