Multimodal transmission of packetized data

• US 10,032,452 B1
• Filed: 12/30/2016
• Issued: 07/24/2018
• Est. Priority Date: 12/30/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bhaya, Gaurav, Stets, Robert
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US15/395,703
• Publication Number:
  US 20180190277A1
• Time in Patent Office: 571 Days
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: H04L 47/25 with rate being modified by the source upon detecting a change of network conditions

Abstract

A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment is provided. A natural language processor component can parse an input audio signal to identify a request and a trigger keyword. Based on the input audio signal, a direct action application programming interface can generate a first action data structure, and a content selector component can select a content item. An interface management component can identify first and second candidate interfaces, and respective resource utilization values. The interface management component can select, based on the resource utilization values, the first candidate interface to present the content item. The interface management component can provide the first action data structure to the client computing device for rendering as audio output, and can transmit the content item converted for a first modality to deliver the content item for rendering from the selected interface.

Claims

1. A system of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:

a natural language processor component executed by a data processing system to receive, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device;

the natural language processor component to parse the input audio signal to identify a request and a trigger keyword corresponding to the request;

a direct action application programming interface of the data processing system to generate, based on at least one of the request and the trigger keyword, a first action data structure;

a content selector component executed by the data processing system to receive at least one of the request and the trigger keyword identified by the natural language processor and to select, based on at least one of the request and the trigger keyword, a content item via a real-time content selection process;

an interface management component of the data processing system to;

poll a plurality of interfaces to identify a first candidate interface and a second candidate interface;

determine a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface, the first resource utilization value and the second resource utilization value based on at least one of a battery status, processor utilization, memory utilization, an interface parameter, and network bandwidth utilization;

select, based on a comparison of the first resource utilization value and the second resource utilization value, the first candidate interface as a selected interface to present the content item;

convert the content item for delivery in a modality compatible with the selected interface selected based on the comparison of the first resource utilization value and the second resource utilization value;

provide the first action data structure to the client computing device for rendering as audio output from the client computing device; and

transmit the content item converted for the first modality to deliver the content item for rendering from the selected interface.

2. The system of claim 1, comprising:

the interface management component to deliver the content item to the client computing device via the selected interface subsequent to transmission of the data structure to the client computing device.

3. The system of claim 1, wherein first candidate interface and second candidate interface are two different interfaces of the client device.

4. The system of claim 3, comprising:

the interface management component to provide the first action data structure for rendering from the second candidate interface.

5. The system of claim 1, wherein the client computing device is a first client computing device, comprising:

the interface management component to determine that the first candidate interface corresponds to the first client computing device linked to an account, and that the second candidate interface corresponds to a second client computing device linked to the account.

6. The system of claim 1, comprising the interface management component to:

determine that the first candidate interface corresponds to an account, and that the second candidate interface corresponds to the account; and

to

transmit a second content item via the second candidate interface based on the first candidate interface and the second candidate interface corresponding to the account.

7. The system of claim 6, wherein the client computing device is a first client computing device, comprising:

the interface management component to determine that the first candidate interface corresponds to the first client computing device and that the second candidate interface corresponds to a second client computing device.

8. The system of claim 1, wherein the first candidate interface includes a display screen, and the second candidate interface includes an audio interface.

9. The system of claim 1, wherein the first candidate interface is associated with a push notification, and the second candidate interface includes a java script slot on an online document.

10. The system of claim 1, comprising the interface management component to:

identify a third candidate interface;

determine a third resource utilization value for the third candidate interface; and

assign a hierarchy to the first candidate interface, the second candidate interface, and the third candidate interface based on the first resource utilization value, the second resource utilization value, and the third resource utilization value.

11. The system of claim 10, comprising:

the interface management component to deliver the content item to the client computing device via the selected interface based on the hierarchy.

12. The system of claim 1, comprising the interface management component to:

determine that the first resource utilization value indicates one of a charging status and a connection to an external power source; and

select the first candidate interface as the selected interface based on the first resource utilization value that indicates one of the charging status and the connection to the external power source.

13. The system of claim 1, wherein the plurality of candidate interfaces include at least one of a display screen, an audio interface, a vibration interface, an email interface, a push notification interface, a mobile computing device interface, a portable computing device application, a content slot on an online document, a chat application, mobile computing device application, a laptop, a watch, a virtual reality headset, and a speaker.

14. The system of claim 1, wherein the client computing device is a first client computing device, comprising the interface management component to:

determine a distance between the first client computing device and a second client computing device; and

to

transmit the content item converted for the first modality to the second client computing device.

15. The system of claim 1, wherein the client computing device is a first client computing device, comprising the interface management component to:

transmit the content item converted for the first modality to the first client computing device within a pre-determined time period from receipt of the input audio signal by the natural language processor component.

16. The system of claim 1, comprising:

the interface management component to determine, from the second resource utilization value, that the second candidate interface is unavailable.

17. The system of claim 1, comprising:

provide the first action data structure to the client computing device for rendering with an indication of the content item, the indication identifying the first candidate interface.

18. A method of multi-modal transmission of packetized data in a voice activated data packet based computer network environment, comprising:

receiving, by a natural language processor component executed by a data processing system, via an interface of the data processing system, data packets comprising an input audio signal detected by a sensor of a client computing device;

identifying, by the natural language processor component, from the input audio signal, a request and a trigger keyword corresponding to the request;

generating, by a direct action application programming interface of the data processing system, based on at least one of and the trigger keyword, a first action data structure;

receiving, by a content selector component executed by the data processing system, at least one of the request and the trigger keyword identified by the natural language processor and selecting, based on at least one of the request and the trigger keyword, a content item via a real-time content selection process;

polling, by an interface management component of the data processing system, a plurality of interfaces to identify a first candidate interface and a second candidate interface;

determining, by the interface management component, a first resource utilization value for the first candidate interface and a second resource utilization value for the second candidate interface, the first resource utilization value and the second resource utilization value based on at least one of a battery status, processor utilization, memory utilization, an interface parameter, and network bandwidth utilization;

selecting, based on a comparison of the first resource utilization value and the second resource utilization value, the first candidate interface as a selected interface to present the content item;

converting the content item for delivery in a modality compatible with the first candidate interface selected based on the comparison of the first resource utilization value and the second resource utilization value;

providing the first action data structure to the client computing device for rendering as audio output from the client computing device; and

transmitting the content item converted for the first modality to deliver the content item for rendering from the selected interface.

19. The method of claim 18, wherein first candidate interface and second candidate interface are two different interfaces of the client device, comprising:

providing the first action data structure for rendering from the second candidate interface.

20. The method of claim 18, wherein the client computing device is a first client computing device, comprising:

transmitting the content item converted for the first modality to the first client computing device within a pre-determined time period from receipt of the input audio signal by the natural language processor component.