Multimodal transmission of packetized data

• US 10,708,313 B2
• Filed: 08/11/2017
• Issued: 07/07/2020
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lewis, Justin, Rapp, Richard, Bhaya, Gaurav, Stets, Robert
• Primary Examiner(s): Jagannathan, Melanie
• Application Number:
  US15/674,838
• Publication Number:
  US 20180191788A1
• Time in Patent Office: 1,061 Days
• CPC Class Codes: G01S 2205/02 Indoor
• CPC Class Codes: G01S 5/02 using radio waves using satellite radio beacon systems for determining position G01S19/00
• CPC Class Codes: G01S 5/0295 Proximity-based methods, e.g. position inferred from reception of particular signals
• CPC Class Codes: G01S 5/18 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: H04L 45/70 Routing based on monitoring results
• CPC Class Codes: H04L 65/1066 Session management
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

Claims

1. A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:

a data processing system comprising hardware circuitry and memory to;

receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a computing device;

parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

generate, based on at least one of the request and the trigger keyword, a first action data structure;

select, based on at least one of the request and the trigger keyword, a content item via a real-time content selection process;

poll a plurality of interfaces to identify a first candidate interface and a second candidate interface within a threshold distance of the computing device;

determine a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface;

select, based on a comparison of the first resource utilization value and the second resource utilization value, the first candidate interface as a selected interface to present the content item;

provide the content item for delivery in a modality compatible with the selected interface selected based on the comparison of the first resource utilization value and the second resource utilization value; and

transmit the content item in the modality to deliver the content item for presentation via the selected interface.

2. The system of claim 1, comprising:

the data processing system to instruct the computing device to transmit a sub-audible wave to identify a second computing device within the threshold distance of the computing device, wherein the second computing device comprises the second candidate interface.

3. The system of claim 1, comprising the data processing system to:

provide a token, via a sub-audible wave transmitted by the computing device, for detection by a second sensor of a second computing device;

receive, from the second computing device via a network, the token detected by the second sensor of the second computing device; and

determine, based on a match between the token received from the second computing device and the token provided via the sub-audible wave transmitted by the computing device, that the second computing device is within the threshold distance of the computing device.

4. The system of claim 1, comprising:

the data processing system to instruct the computing device to transmit a short-range wireless transmission comprising a signal to identify a second computing device within the threshold distance of the computing device, wherein the second computing device comprises the second candidate interface.

5. The system of claim 4, comprising the data processing system to:

compare the input audio signal detected by the sensor of the computing device with a second input audio signal detected by a second sensor of a second computing device to identify a match; and

identify, based on the match, the second computing device as within the threshold distance of the computing device, wherein the second computing device comprises the second candidate interface.

6. The system of claim 5, comprising the data processing system to:

receive a first timestamp corresponding to transmission, by the computing device, of a sub-audible wave comprising the token;

receive a second timestamp corresponding to detection, by the second computing device, of the sub-audible wave comprising the token; and

determine, based on a difference between the second timestamp and the first timestamp, that the second computing device is within the threshold distance from the computing device.

7. The system of claim 1, comprising the data processing system to:

identify, based on a plurality of proximity detection techniques, a plurality of computing devices within the threshold distance of the computing device, the plurality of computing devices comprising a second computing device comprising the second candidate interface and a third computing device comprising a third candidate interface;

poll the plurality of interfaces comprising the first candidate interface, the second candidate interface, and the third candidate interface;

determine a third resource utilization value for the third candidate interface; and

select, based on a comparison of the first resource utilization value, the second resource utilization value, and the third resource utilization value, the first candidate interface as the selected interface to present the content item.

8. The system of claim 1, comprising:

the data processing system to retrieve, from memory, proximity information identifying a plurality of computing devices within the threshold distance of the computing device.

9. The system of claim 1, comprising the data processing system to:

identify a second computing device comprising the second candidate interface;

query an application programming interface executing on the second computing device to determine the second resource utilization value, the second resource utilization value indicating a volume level at which audio content is played.

10. The system of claim 1, wherein the first resource utilization value and the second resource utilization value are determined based on at least one of a battery status, processor utilization, memory utilization, an interface parameter, and network bandwidth utilization.

11. The system of claim 1, wherein the first resource utilization value and the second resource utilization value are determined based on sensor utilization.

12. The system of claim 1, wherein the first candidate interface and the second candidate interface are two different interfaces of the computing device.

13. The system of claim 1, wherein the computing device is a first client computing device, comprising:

the data processing system to determine that the first candidate interface corresponds to the first client computing device linked to an account, and that the second candidate interface corresponds to a second computing device linked to the account.

14. The system of claim 1, wherein the first candidate interface includes a display screen, and the second candidate interface includes an audio interface.

15. The system of claim 1, wherein the first candidate interface is associated with a push notification, and the second candidate interface includes a java script slot on an online document.

16. The system of claim 1, comprising the data processing system to:

identify a third candidate interface;

determine a third resource utilization value for the third candidate interface; and

assign a hierarchy to the first candidate interface, the second candidate interface, and the third candidate interface based on the first resource utilization value, the second resource utilization value, and the third resource utilization value.

17. The system of claim 1, wherein the plurality of interfaces include at least one of a display screen, an audio interface, a vibration interface, an email interface, a push notification interface, a mobile computing device interface, a portable computing device application, a content slot on an online document, a chat application, mobile computing device application, a laptop, a watch, a virtual reality headset, and a speaker.

18. A method of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:

receiving, by a data processing system comprising hardware circuitry and memory, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a computing device;

identifying, by the data processing system, from the input audio signal, a request and a trigger keyword corresponding to the request;

generating, by the data processing system, based on at least one of and the trigger keyword, a first action data structure;

selecting, based on at least one of the request and the trigger keyword, a content item via a real-time content selection process;

polling, by the data processing system, a plurality of interfaces to identify a first candidate interface and a second candidate interface within a threshold distance of the computing device;

determining, by the data processing system, a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface;

selecting, based on a comparison of the first resource utilization value and the second resource utilization value, the first candidate interface as a selected interface to present the content item;

converting the content item for delivery in a modality compatible with the first candidate interface selected based on the comparison of the first resource utilization value and the second resource utilization value; and

transmitting the content item converted for the modality to deliver the content item for rendering from the selected interface.

19. The method of claim 18, comprising:

instructing, by the data processing system, the computing device to transmit a sub-audible wave to identify a second computing device within the threshold distance of the computing device, wherein the second computing device comprises the second candidate interface.

20. The method of claim 18, comprising:

providing a token, via a sub-audible wave transmitted by the computing device, for detection by a second sensor of a second computing device;

receiving, from the second computing device via a network, the token detected by the second sensor of the second computing device; and

determining, based on a match between the token received from the second computing device and the token provided via the sub-audible wave transmitted by the computing device, that the second computing device is within the threshold distance of the computing device.