Modulation of packetized audio signals

• US 10,347,247 B2
• Filed: 12/30/2016
• Issued: 07/09/2019
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bhaya, Gaurav, Stets, Robert
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US15/395,660
• Publication Number:
  US 20180190275A1
• Time in Patent Office: 921 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/162 Interface to dedicated audio devices, e.g. audio drivers, interface to CODECs
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 13/027 Concept to speech synthesisers; Generation of natural phrases from machine-based concepts generation of parameters for speech synthesis out of text G10L13/08
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 21/003 Changing voice quality, e.g. pitch or formants
• CPC Class Codes: G10L 21/0316 by changing the amplitude
• CPC Class Codes: H04L 65/1069 Session establishment or de-establishment
• CPC Class Codes: H04N 21/440236 by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859

Abstract

Modulating packetized audio signals in a voice activated data packet based computer network environment is provided. A system can receive audio signals detected by a microphone of a device. The system can parse the audio signal to identify trigger keyword and request, and generate a first action data structure. The system can identify a content item object based on the trigger keyword, and generate an output signal comprising a first portion corresponding to the first action data structure and a second portion corresponding to the content item object. The system can apply a modulation to the first or second portion of the output signal, and transmit the modulated output signal to the device.

Claims

1. A system to modulate packetized audio signals, comprising:

a natural language processor component executed by a data processing system to receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client device;

the natural language processor component to parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

a session handler component executed by the data processing system to establish a communication session between the client device and the data processing system, the communication session comprising the input audio signal;

a direct action application programming interface of the data processing system to generate, based on the trigger keyword, a first action data structure responsive to the request, the first action data structure corresponding to a first type of content;

a content selector component executed by the data processing system to receive the trigger keyword identified by the natural language processor component and select, based on the trigger keyword, a content item via a real-time content selection process, the content item corresponding to a second type of content different from the first type of content;

an audio signal generator component executed by the data processing system to generate an output signal comprising a first portion corresponding to the first action data structure and generated with a first audio modulation, and a second portion corresponding to the content item and generated with a second audio modulation different from the first audio modulation;

the interface of the data processing system to transmit data packets comprising the output signal generated by the audio signal generator component to cause an audio driver component executed by the client device to drive a speaker of the client device to generate an acoustic wave corresponding to the output signal;

the content selector component to select, for the communication session comprising the input audio signal, an additional content item via the real-time content selection process, the additional content item corresponding to the second type of content different from the first type of content;

a policy engine component executed by the data processing system to;

determine that the additional content item corresponds to the content item previously selected via the real-time content selection process during the communication session;

instruct, responsive to the determination that the additional content item corresponds to the content item previously selected during the communication session, the audio signal generator component to generate an additional output signal comprising an additional portion corresponding to the additional content item of the second type of content selected by the content selector component, the additional portion generated with the first audio modulation previously used to generate the first portion corresponding to the first action data structure instead of the second audio modulation previously used to generate the second portion corresponding to the content item of the second type of content; and

the interface to transmit, within the communication session and to the client device, data packets comprising the additional output signal.

2. The system of claim 1, comprising:

the natural language processor component to apply a semantic processing technique to the input audio signal to identify a trigger phrase comprising the trigger keyword and a second trigger keyword.

3. The system of claim 1, comprising:

the natural language processor component to filter the input audio signal to identify the trigger keyword.

4. The system of claim 1, comprising:

the policy engine component executed by the data processing system to determine, based on the content item selected via the real-time content selection process, to instruct the audio signal generator component to use the second audio modulation different from the first audio modulation.

5. The system of claim 1, comprising:

the session handler component executed by the data processing system to establish the communication session between the client device and the data processing system, the communication session comprising the input audio signal; and

the policy engine component executed by the data processing system to;

determine that the content item selected via the real-time content selection process was not previously presented to the client device during the communication session; and

instruct, responsive to the content item not previously provided during the communication session, the audio signal generator component to generate the second portion of the output signal with the second audio modulation different from the first audio modulation.

6. The system of claim 1, comprising:

the session handler component executed by the data processing system to establish the communication session between the client device and the data processing system, the communication session comprising the input audio signal and a second input audio signal; and

the content selector component to select, based on the second input audio signal, a second content item different from one or more content items previously selected and provided for the client device during the communication session;

the policy engine component executed by the data processing system to;

determine that the second content item is different from the one or more content items previously selected and provided for the client device during the communication session;

instruct, responsive to the second content item different from the one or more content items previously provided during the communication session, the audio signal generator component to generate a second output signal comprising a third portion corresponding to the second content item selected by the content selector component, the third portion generated with the second audio modulation; and

the interface to transmit, within the communication session and to the client device, data packets comprising the second output signal.

7. The system of claim 1, comprising:

the session handler component executed by the data processing system to establish the communication session between the client device and the data processing system, the communication session comprising the input audio signal;

the policy engine component executed by the data processing system to;

determine that the real-time content selection process previously selected and provided one or more content items for the client device during the communication session; and

instruct, responsive to the one or more content items previously provided during the communication session, the audio signal generator component to generate a second output signal comprising a third portion corresponding to a second content item selected by the content selector component, the third portion generated with the first audio modulation; and

the interface to transmit, within the communication session and to the client device, data packets comprising the second output signal.

8. The system of claim 7, comprising:

the session handler component executed by the data processing system to;

detect, subsequent to transmission of the data packets comprising the second output signal, a transition from a first type of network activity that occurred in the communication session to a second type of network activity;

terminate, responsive to the transition, the communication session; and

establish a second communication session between the client device and the data processing system, the second communication session comprising the second type of network activity and a second input audio signal;

the policy engine component to determine that at least one content item was not previously provided in the second communication session; and

the audio signal generator component to generate a third output signal comprising a fourth portion corresponding to a third content item selected by the content selector component, the fourth portion generated with the second audio modulation responsive to the determination that at least one content item was not previously provided in the second communication session.

9. The system of claim 1, comprising:

the audio signal generator component to generate the second audio modulation comprising at least one of a frequency shift and an amplitude shift.

10. The system of claim 1, comprising:

the audio signal generator component to generate the second audio modulation comprising at least one of an audio overlay and an audio watermark.

11. The system of claim 1, comprising:

the audio signal generator component to generate the first portion with the first audio modulation comprising a default audio modulation, generate the second portion with the second audio modulation, wherein the second audio modulation comprises a frequency or amplitude greater than a frequency or amplitude of the default audio modulation.

12. A method of performing dynamic modulation of packetized audio signals, comprising:

receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client device;

parsing, by the natural language processor component, the input audio signal to identify a request and a trigger keyword corresponding to the request;

establishing, by a session handler component executed by the data processing system, a communication session between the client device and the data processing system, the communication session comprising the input audio signal;

generating, by a direct action application programming interface of the data processing system, based on the trigger keyword, a first action data structure responsive to the request, the first action data structure corresponding to a first type of content;

receiving, by a content selector component executed by the data processing system, the trigger keyword identified by the natural language processor component;

selecting, by the content selector component based on the trigger keyword, a content item via a real-time content selection process, the content item corresponding to a second type of content different from the first type of content;

generating, by an audio signal generator component executed by the data processing system, an output signal comprising a first portion corresponding to the first action data structure and generated with a first audio modulation, and a second portion corresponding to the content item and generated with a second audio modulation different from the first audio modulation;

transmitting, by the interface of the data processing system, data packets comprising the output signal generated by the audio signal generator component to cause an audio driver component executed by the client device to drive a speaker of the client device to generate an acoustic wave corresponding to the output signal;

selecting, by the content selector component, for the communication session comprising the input audio signal, an additional content item via the real-time content selection process, the additional content item corresponding to the second type of content different from the first type of content;

determining, by the content selector component, that the additional content item corresponds to the content item previously selected via the real-time content selection process during the communication session;

instructing, by the content selector component, responsive to the determination that the additional content item corresponds to the content item previously selected during the communication session, the audio signal generator component to generate an additional output signal comprising an additional portion corresponding to the additional content item of the second type of content selected by the content selector component, the additional portion generated with the first audio modulation previously used to generate the first portion corresponding to the first action data structure instead of the second audio modulation previously used to generate the second portion corresponding to the content item of the second type of content; and

transmitting, by the data processing system, within the communication session and to the client device, data packets comprising the additional output signal.

13. The method of claim 12, comprising:

applying, by the data processing system, a semantic processing technique to the input audio signal to identify a trigger phrase comprising the trigger keyword and a second trigger keyword.

14. The method of claim 12, comprising:

filtering, by the data processing system, the input audio signal to identify the trigger keyword.

15. The method of claim 12, comprising:

determining, by the data processing system, based on the content item selected via the real-time content selection process, to instruct the audio signal generator component to use the second audio modulation different from the first audio modulation.

16. The method of claim 12, comprising:

establishing, by the data processing system, the communication session between the client device and the data processing system, the communication session comprising the input audio signal;

determining, by the data processing system, that the content item selected via the real-time content selection process was not previously presented to the client device during the communication session; and

instructing, by the data processing system, responsive to the content item not previously provided during the communication session, the audio signal generator component to generate the second portion of the output signal with the second audio modulation different from the first audio modulation.

17. The method of claim 12, comprising:

establishing, by the data processing system, the communication session between the client device and the data processing system, the communication session comprising the input audio signal and a second input audio signal;

selecting, by the data processing system, based on the second input audio signal, a second content item different from one or more content items previously selected and provided for the client device during the communication session;

determining, by the data processing system, that the second content item is different from the one or more content items previously selected and provided for the client device during the communication session;

instructing, by the data processing system, responsive to the second content item different from the one or more content items previously provided during the communication session, the audio signal generator component to generate a second output signal comprising a third portion corresponding to the second content item selected by the content selector component, the third portion generated with the second audio modulation; and

transmitting, by the data processing system to the client device during the communication session, data packets comprising the second output signal.

18. The method of claim 12, comprising:

establishing, by the data processing system, the communication session between the client device and the data processing system, the communication session comprising the input audio signal;

determining, by the data processing system, that the real-time content selection process previously selected and provided one or more content items for the client device during the communication session;

instructing, by the data processing system, responsive to the one or more content items previously provided during the communication session, the audio signal generator component to generate a second output signal comprising a third portion corresponding to a second content item selected by the content selector component, the third portion generated with the first audio modulation; and

transmitting, by the data processing system to the client device, during the communication session, data packets comprising the second output signal.

19. The method of claim 18, comprising:

detecting, by the data processing system subsequent to transmission of the data packets comprising the second output signal, a transition from a first type of network activity that occurred in the communication session to a second type of network activity;

terminating, by the data processing system responsive to the transition, the communication session;

establishing, by the data processing system, a second communication session between the client device and the data processing system, the second communication session comprising the second type of network activity and a second input audio signal;

determining, by the data processing system, that at least one content item was not previously provided in the second communication session; and

generating, by the data processing system, a third output signal comprising a fourth portion corresponding to a third content item selected by the content selector component, the fourth portion generated with the second audio modulation responsive to the determination that at least one content item was not previously provided in the second communication session.

20. The method of claim 12, comprising:

generating, by the data processing system, the second audio modulation comprising at least one of a frequency shift and an amplitude shift.