Multimodal transmission of packetized data

• US 10,593,329 B2
• Filed: 07/18/2018
• Issued: 03/17/2020
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bhaya, Gaurav, Stets, Robert, Patil, Umesh
• Primary Examiner(s): Guerra-Erazo, Edgar X
• Application Number:
  US16/039,204
• Publication Number:
  US 20180322879A1
• Time in Patent Office: 608 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

Claims

1. A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:

a natural language processor component executed by a data processing system to receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device;

the natural language processor component to parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

a direct action application programming interface of the data processing system to generate, based on at least one of the request and the trigger keyword, one or more action data structures;

a content selector component executed by the data processing system to receive at least one of the request and the trigger keyword identified by the natural language processor component and to select, based on at least one of the request and the trigger keyword, one or more content items via a real-time content selection process;

an interface management component of the data processing system to;

poll a plurality of interfaces to identify a first candidate interface and a second candidate interface within a threshold distance of the client computing device;

determine a first device resource utilization value for the first candidate interface and a second device resource utilization value for the second candidate interface;

select, based on a comparison of the first device resource utilization value and the second device resource utilization value, the first candidate interface as a selected interface to present a first action data structure of the one or more action data structures and a first content item of the one or more content items;

provide the first action data structure and the first content item for delivery in a modality compatible with the selected interface selected based on the comparison of the first device resource utilization value and the second device resource utilization value; and

transmit the first action data structure and the first content item in the modality to deliver the content item for presentation via the selected interface.

2. The system of claim 1, wherein the one or more action data structures includes the first action data structure and a second action data structure, comprising:

the data processing system to select a second computing device comprising the second candidate interface to present the second action data structure.

3. The system of claim 1, comprising:

the data processing system to select the first candidate interface based on a category of the first action data structure and a type of the first candidate interface.

4. The system of claim 1, comprising:

the data processing system to select the first candidate interface based on a category of the first action data structure corresponding to a type of the first candidate interface in a map data structure.

5. The system of claim 1, comprising:

the data processing system to select the first candidate interface based on an output type of the first action data structure corresponding to a type of the first candidate interface in a map data structure.

6. The system of claim 1, wherein the one or more action data structures comprises the first action data structure and a second action data structure, comprising the data processing system to:

determine a second content item for the second action data structure; and

select the second candidate interface provided via a second candidate device to present the second content item.

7. The system of claim 1, comprising the data processing system to:

select a second content item for the first action data structure; and

transmit the second content item to the second candidate interface provided via a second candidate device.

8. The system of claim 1, comprising the data processing system to:

split the first action data structure into a plurality of action data structures; and

transmit the plurality of action data structures to a plurality of candidate interfaces for presentation.

9. The system of claim 1, comprising the data processing system to:

receive a request to present the first action data structure via the second candidate interface;

determine to override the request; and

select the first candidate interface to present the first action data structure responsive to the determination to override the request.

10. The system of claim 1, comprising the data processing system to:

receive a request to present the first action data structure via the second candidate interface;

determine that a category of the first action data structure corresponds to the first candidate interface in a map data structure;

override the request responsive to the determination based on the map data structure; and

select the first candidate interface to present the first action data structure responsive to the determination to override the request.

11. The system of claim 1, comprising the data processing system to:

receive a request to present the first action data structure via the second candidate interface;

determine that an output type of the first content item selected for the first action data structure corresponds to the first candidate interface in a map data structure;

override the request responsive to the determination based on the map data structure; and

select the first candidate interface to present the first action data structure and the first content item responsive to the determination to override the request.

12. The system of claim 1, wherein the first device resource utilization value comprises at least one of a battery status, processor utilization, memory utilization, an interface parameter, and network bandwidth utilization associated with the first candidate interface, and the second device resource utilization value comprises at least one of a battery status, processor utilization, memory utilization, an interface parameter, and network bandwidth utilization associated with the second candidate interface.

13. The system of claim 1, comprising the data processing system to:

receive a request to present the first action data structure via the second candidate interface;

determine a category or an output type of the first action data structure;

override the request based on historical interfaces used to present action data structures having the category or the output type; and

select the first candidate interface to present the first action data structure responsive to the determination to override the request.

14. A method of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:

receiving, by a data processing system comprising one or more processors and memory, data packets comprising an input audio signal detected by a sensor of a client computing device;

parsing, by the data processing system, the input audio signal to identify a request and a trigger keyword corresponding to the request;

generating, by the data processing system, based on at least one of the request and the trigger keyword, one or more action data structures;

receiving, by the data processing system, at least one of the request and the trigger keyword identified by a natural language processor;

selecting, by the data processing system based on at least one of the request and the trigger keyword, one or more content items via a real-time content selection process;

polling, by the data processing system, a plurality of interfaces to identify a first candidate interface and a second candidate interface within a threshold distance of the client computing device;

determining, by the data processing system, a first device resource utilization value for the first candidate interface and a second device resource utilization value for the second candidate interface;

selecting, by the data processing system, based on a comparison of the first device resource utilization value and the second device resource utilization value, the first candidate interface as a selected interface to present a first action data structure of the one or more action data structures and a first content item of the one or more content items;

providing, by the data processing system, the first action data structure and the first content item for delivery in a modality compatible with the selected interface selected based on the comparison of the first device resource utilization value and the second device resource utilization value; and

transmitting, by the data processing system, the first action data structure and the first content item in the modality to deliver the content item for presentation via the selected interface.

15. The method of claim 14, wherein the one or more action data structures includes the first action data structure and a second action data structure, comprising:

selecting a second computing device comprising the second candidate interface to present the second action data structure.

16. The method of claim 14, comprising:

selecting the first candidate interface based on a category of the first action data structure and a type of the first candidate interface.

17. The method of claim 14, comprising:

selecting the first candidate interface based on a category of the first action data structure corresponding to a type of the first candidate interface in a map data structure.

18. The method of claim 14, comprising:

selecting the first candidate interface based on an output type of the first action data structure corresponding to a type of the first candidate interface in a map data structure.

19. The method of claim 14, wherein the one or more action data structures comprises the first action data structure and a second action data structure, comprising:

determining a second content item for the second action data structure; and

selecting the second candidate interface provided via a second candidate device to present the second content item.

20. The method of claim 14, comprising:

selecting a second content item for the first action data structure; and

transmitting the second content item to the second candidate interface provided via a second candidate device.