PROACTIVE ASSISTANCE BASED ON DIALOG COMMUNICATION BETWEEN DEVICES

• US 20170185375A1
• Filed: 05/31/2016
• Published: 06/29/2017
• Est. Priority Date: 12/23/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: MARTEL, Mathieu Jean, DENIAU, Thomas
• Granted Patent: US 10,223,066 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Systems and processes for proactive assistance based on dialog communication between devices are provided. In one example process, while voice communication between an electronic device and a second electronic device is established, a stream of audio data associated with the second electronic device can be received. In response to detecting a user input, a text representation of speech contained in a portion of the stream of audio data can be generated. The process can determine whether the text representation contains information corresponding to one of a plurality of types of information. In response to determining that the text representation contains information corresponding to one of a plurality of types of information, one or more tasks based on the information can be performed.

Claims

1. A non-transitory computer-readable medium storing instructions for providing proactive assistance based on dialog communication between devices, the instructions, when executed by one or more processors, cause the one or more processors to:

while voice communication is established between an electronic device and a second electronic device;

receive a stream of audio data associated with the second electronic device;

detect a user input;

in response to detecting the user input, generate a text representation of speech contained in a portion of the stream of audio data;

determine whether the text representation contains information corresponding to one of a plurality of types of information; and

in response to determining that the text representation contains information corresponding to one of a plurality of types of information, perform one or more tasks based on the information.

2. The computer-readable medium of claim 1, wherein the user input is detected by a physical sensor of the electronic device.

3. The computer-readable medium of claim 1, wherein a proximity sensor of the electronic device is positioned less than a threshold distance from a surface, and wherein the user input includes changing a position of the electronic device such that the proximity sensor is positioned greater than the threshold distance from the surface.

4. The computer-readable medium of claim 1, wherein the user input is detected by an accelerometer of the electronic device, and wherein the user input includes positioning the electronic device for a user to view a display of the electronic device.

5. The computer-readable medium of claim 1, wherein the user input include selecting an affordance on the electronic device.

6. The computer-readable medium of claim 1, wherein detecting the user input causes a user interface to be displayed on a display of the electronic device, and wherein the instructions further cause the one or more processors to display the information via the user interface.

7. The computer-readable medium of claim 1, wherein the user input includes a user utterance, and wherein detecting the user input includes detecting one or more trigger words in the user utterance.

8. The computer-readable medium of claim 1, wherein the instructions further cause the one or more processors to:

determine whether the information is complete;

in response to determining that the information is not complete, generate a text representation of speech contained in a second portion of the stream of audio data that is different from the first portion of the stream of audio data; and

obtain second information from the second portion of the stream of audio data, wherein the one or more tasks are based on the second information.

9. The computer-readable medium of claim 1, wherein the information includes a telephone number, wherein the one or more tasks include displaying the telephone number, and wherein the instructions further cause the one or more processors to:

in response to detecting a user selection of the displayed telephone number, initiate a voice call based on the telephone number.

10. The computer-readable medium of claim 1, wherein the information includes a telephone number, wherein the one or more tasks include displaying the telephone number, and wherein the instructions further cause the one or more processors to:

in response to detecting a user selection of the displayed telephone number, store the telephone number in association with an address book of the electronic device.

11. The computer-readable medium of claim 1, wherein the information includes an email address, wherein the one or more tasks include displaying the email address, and wherein the instructions further cause the one or more processors to:

in response to detecting a user selection of the displayed email address, initiate a composition of an email message, wherein a recipient of the email message is based on the email address.

12. The computer-readable medium of claim 1, wherein the information includes a location, and wherein the one or more tasks include displaying a map indicating the location.

13. A non-transitory computer-readable medium storing instructions for providing proactive assistance based on dialog communication between devices, the instructions, when executed by one or more processors, cause the one or more processors to:

receive text associated with a second electronic device;

detect a user input;

in response to detecting the user input, determine whether a portion of the text contains information corresponding to one of a plurality of types of information; and

in response to determining that the portion of the text contains information corresponding to one of a plurality of types of information, perform one or more tasks based on the information.

14. The computer-readable medium of claim 13, wherein the user input defines the portion of the text.

15. The computer-readable medium of claim 13, wherein detecting the user input causes a user interface to be displayed on a display of the electronic device, and wherein the one or more tasks include displaying the information via the user interface.

16. The computer-readable medium of claim 13, wherein the one or more tasks include:

storing the information;

providing an indicia associated with the information;

in response to detecting a user selection associated with the indicia, displaying the information on the electronic device.

17. The computer-readable medium of claim 13, wherein the information includes a telephone number, wherein the one or more tasks include displaying the telephone number, and wherein the instructions further cause the one or more processors to:

in response to detecting a user selection of the displayed telephone number, initiate a voice call based on the telephone number.

18. The computer-readable medium of claim 13, wherein the information includes a telephone number, wherein the one or more tasks include displaying the telephone number, and wherein the instructions further cause the one or more processors to:

in response to detecting a user selection of the displayed telephone number, store the telephone number in association with an address book of the electronic device.

19. A method for providing proactive assistance based on dialog communication between devices, the method comprising:

at an electronic device;

while voice communication is established between the electronic device and a second electronic device;

receiving a stream of audio data associated with the second electronic device;

detecting a user input;

in response to detecting the user input, generating a text representation of speech contained in a portion of the stream of audio data;

determining whether the text representation contains information corresponding to one of a plurality of types of information; and

in response to determining that the text representation contains information corresponding to one of a plurality of types of information, performing one or more tasks based on the information.

20. An electronic device for providing proactive assistance based on dialog communication between devices, the electronic device comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the one or more processors to;

while voice communication is established between the electronic device and a second electronic device;

receive a stream of audio data associated with the second electronic device;

detect a user input;

in response to detecting the user input, generate a text representation of speech contained in a portion of the stream of audio data;

determine whether the text representation contains information corresponding to one of a plurality of types of information; and

in response to determining that the text representation contains information corresponding to one of a plurality of types of information, perform one or more tasks based on the information.