Sequence dependent operation processing of packet based data message transmissions

• US 10,854,198 B2
• Filed: 06/26/2018
• Issued: 12/01/2020
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bhaya, Gaurav, Stets, Robert
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US16/018,854
• Publication Number:
  US 20180308484A1
• Time in Patent Office: 889 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Optimization of sequence dependent operations in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. A prediction component can determine a thread based on the trigger keyword and the request that includes a first action, a second action subsequent to the first action, and a third action subsequent to the second action. A content selector component can select, based on the third action and the trigger keyword, a content item. An audio signal generator component can generate an output signal comprising the content item. An interface can transmit the output signal to cause a client computing device to drive a speaker to generate an acoustic wave corresponding to the output signal prior to occurrence of at least one of the first action and the second action.

Claims

1. A system to process sequence dependent operations in a voice activated data packet based computer network environment, comprising:

a natural language processor component executed by a data processing system to receive input data packets comprising an input audio signal detected by a sensor of a client computing device;

the natural language processor component to parse the input audio signal to identify a keyword;

a prediction component to determine a thread based on the keyword, the thread comprising a first action, a second action subsequent to the first action, and a third action subsequent to the second action;

the prediction component to provide, to a content selector component of the data processing system, an indication of the third action prior to occurrence of at least one of the first action and the second action;

the content selector component to bypass content selection based on the first action and the second action in order to identify, via a real-time content selection process, a content item based on the third action predicted to occur by the prediction component, the content item provided by a third-party content provider;

an interface of the data processing system to transmit data packets comprising the content item to cause the client computing device to present the content item;

the content selector component to identify, via a subsequent real-time content selection process executed subsequent to occurrence of the first action, a second content item provided by the third-party content provider or a second third-party content provider; and

the interface to transmit the second content item to cause the client computing device to present the second content item prior to occurrence of the third action.

2. The system of claim 1, comprising:

the natural language processor component to parse the input audio signal to identify a request and a trigger keyword corresponding to the request.

3. The system of claim 1, comprising the data processing system to:

generate an output signal comprising the content item; and

transmit the output signal to cause an audio driver component executed by the client computing device to drive a speaker of the client computing device to generate an acoustic wave corresponding to the output signal prior to occurrence of at least one of the first action and the second action.

4. The system of claim 1, comprising the data processing system to:

generate an output signal comprising the content item; and

transmit the data packets comprising the output signal to cause an audio driver component executed by the client computing device to drive a speaker of the client computing device to generate an acoustic wave corresponding to the output signal subsequent to occurrence of the first action and prior to occurrence of the second action.

5. The system of claim 1, comprising the data processing system to:

the data processing system to receive, from the client computing device a data message that includes a response to the content item prior to occurrence of at least one of the first action or the second action.

6. The system of claim 1, comprising:

the data processing system to initiate the third action based on one of a time of completion of the first action and a time of completion of the second action.

7. The system of claim 1, comprising:

a direct action application programming interface of the data processing system to initiate the third action subsequent to completion of at least one of the first action and the second action.

8. A system to expand sequence dependent operations in a voice activated data packet based computer network environment, comprising:

a natural language processor component executed by a data processing system to receive input data packets comprising an input audio signal detected by a sensor of a client computing device;

the natural language processor component to parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

the data processing system to determine a thread based on the trigger keyword and the request, the thread comprising a first action, a second action subsequent to the first action, and a third action subsequent to the second action;

an audio signal generator component executed by the data processing system to generate an output signal corresponding to the thread;

an interface of the data processing system to transmit data packets comprising the output signal generated by the audio signal generator component and cause an audio driver component executed by the client computing device to drive a speaker of the client computing device to generate an acoustic wave corresponding to the output signal prior to occurrence of at least one of the first action, the second action or the third action;

a content selector component of the data processing system to bypass content selection based on the first action and the second action in order to identify, via a real-time content selection process executed prior to occurrence of the third action, a content item based on the third action, the content item provided by a third-party content provider; and

transmit the content item to cause the client computing device to present the content item prior to occurrence of the third action.

9. The system of claim 8, comprising the data processing system to:

transmit the data packets comprising the output signal to cause to cause the audio driver component executed by the client computing device to drive the speaker of the client computing device to generate the acoustic wave corresponding to the output signal subsequent to occurrence of the first action and prior to occurrence of the second action.

10. The system of claim 8, comprising the data processing system to:

the data processing system to receive, from the client computing device a data message that includes a response to the content item prior to occurrence of at least one of the first action or the second action.

11. The system of claim 8, comprising:

the data processing system to initiate the third action based on one of a time of completion of the first action and a time of completion of the second action.

12. The system of claim 8, comprising:

a direct action application programming interface of the data processing system to initiate the third action subsequent to completion of at least one of the first action and the second action.

13. The system of claim 8, comprising:

a direct action application programming interface of the data processing system to initiate activity corresponding to the third action and to subsequently initiate activity corresponding to the second action.

14. The system of claim 8, comprising the data processing system to:

select, for the first action, a second content item via a second real-time content selection process and to select, for the second action, a third content item via a third real-time selection process.

15. A method of processing sequence dependent operations in a voice activated data packet based computer network environment, comprising:

receiving, by a data processing system comprising one or more processors and memory, data packets comprising an input audio signal detected by a sensor of a client computing device;

parsing, by the data processing system, the input audio signal to identify a keyword;

determining, by the data processing system, a thread based on the keyword, the thread comprising a first action, a second action subsequent to the first action, and a third action subsequent to the second action;

providing, by the data processing system, an indication of the third action prior to occurrence of at least one of the first action and the second action;

bypassing, by the data processing system, content selection based on the first action and the second action in order to identify a content item based on the third action predicted to occur by the data processing system, the content item provide by a third-party content provider;

transmitting, by the data processing system, data packets comprising the content item to cause the client computing device to present the content item;

identifying, by the data processing system via a real-time content selection process executed subsequent to the occurrence of the first action, a second content item provided by the third-party content provider or a second third-party content provider; and

transmitting, by the data processing system, the second content item to cause the client computing device to present the second content item prior to occurrence of the third action.

16. The method of claim 15, comprising:

identifying the content item via a second real-time content selection process.