Automatically adapting user interfaces for hands-free interaction

• US 10,705,794 B2
• Filed: 06/08/2013
• Issued: 07/07/2020
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gruber, Thomas R., Saddler, Harry J., Napolitano, Lia T., Schubert, Emily Clark, Sumner, Brian Conrad
• Primary Examiner(s): Silverman, Seth A
• Application Number:
  US13/913,421
• Publication Number:
  US 20130275875A1
• Time in Patent Office: 2,586 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks

Abstract

The method includes automatically, without user input and without regard to whether a digital assistant application has been separately invoked by a user, determining that the electronic device is in a vehicle. In some implementations, determining that the electronic device is in a vehicle comprises detecting that the electronic device is in communication with the vehicle (e.g., via a wired or wireless communication techniques and/or protocols). The method also includes, responsive to the determining, invoking a listening mode of a virtual assistant implemented by the electronic device. In some implementations, the method also includes limiting the ability of a user to view visual output presented by the electronic device, provide typed input to the electronic device, and the like.

Claims

1. A method of adapting a user interface, performed at an electronic device having one or more processors and memory storing one or more programs for execution by the one or more processors, the method comprising:

automatically, without user input and without regard to whether a digital assistant application has been separately invoked by a user, determining that the electronic device is in a vehicle; and

responsive to the determining;

automatically invoking a listening mode of a virtual assistant implemented by the electronic device, wherein the listening mode causes the electronic device to automatically, without a physical input from the user, listen for voice input from the user for a predetermined time after the electronic device provides an auditory output;

receiving a voice input from the user; and

concurrently providing both an auditory output and a visual output on a display screen of the electronic device based on the voice input, wherein at least one of the auditory output and the visual output paraphrases the voice input.

2. The method of claim 1, further comprising:

while in the listening mode, detecting a wake-up word spoken by the user;

in response to detecting the wake-up word, listening for a second voice input from the user;

receiving the second voice input from the user; and

generating a response to the second voice input.

3. The method of claim 1, wherein determining that the electronic device is in a vehicle comprises detecting that the electronic device is in communication with the vehicle.

4. The method of claim 3, wherein the communication is BLUETOOTH®

wireless communication.

5. The method of claim 1, wherein determining that the electronic device is in a vehicle comprises detecting that the electronic device is moving at or above a first predetermined speed.

6. The method of claim 1, further comprising, responsive to the determining, limiting the ability to view visual output presented by the electronic device.

7. The method of claim 1, further comprising, responsive to the determining, limiting the ability to interact with a graphical user interface presented by the electronic device.

8. The method of claim 1, further comprising, responsive to the determining, limiting the ability to use a keyboard on the electronic device.

9. The method of claim 1, further comprising, responsive to the determining, limiting the device so as to not request touch input from the user.

10. The method of claim 1, further comprising, responsive to the determining, limiting the device so as to not respond to touch input from the user.

11. The method of claim 1, further comprising:

generating a response to the voice input, the response including a list of information items to be presented to the user; and

outputting the information items via an auditory output mode, wherein in accordance with a determination that the electronic device is not in a vehicle, the information items are only presented on a display screen of the electronic device.

12. The method of claim 11, further comprising, responsive to the determining that the electronic device is in a vehicle, limiting the amount of items in the list to a predetermined amount.

13. The method of claim 1, wherein the voice input corresponds to content to be sent to a recipient, further comprising:

producing a message corresponding to the voice input, wherein the auditory output and the visual output include the message; and

requesting confirmation prior to sending the message to the recipient.

14. The method of claim 13, wherein requesting confirmation comprises asking the user, via the auditory output mode, whether the text should be sent to the recipient.

15. The method of claim 1, further comprising:

receiving a voice input comprising natural language information;

generating, based on the voice input, a plurality of candidate interpretations of user intent;

inferring a user intent based on the plurality of candidate interpretations of the user intent;

identifying a task based at least in part on the inferred user intent;

executing the identified task; and

in accordance with the execution of the identified task, generating a response to the user.

16. The method of claim 1, wherein both the auditory output and the visual output paraphrase the voice input.

17. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

automatically, without user input and without regard to whether a digital assistant application has been separately invoked by a user, determining that the electronic device is in a vehicle; and

responsive to the determining;

automatically invoking a listening mode of a virtual assistant implemented by the electronic device, wherein the listening mode causes the electronic device to automatically, without a physical input from the user, listen for voice input from the user for a predetermined time after the electronic device provides an auditory output;

receiving a voice input from the user; and

concurrently providing both an auditory output and a visual output on a display screen of the electronic device based on the voice input, wherein at least one of the auditory output and the visual output paraphrases the voice input.

18. The non-transitory computer-readable medium of claim 17, wherein determining that the electronic device is in a vehicle comprises detecting that the electronic device is in communication with the vehicle.

19. The non-transitory computer-readable medium of claim 17, the instructions further causing the processors to perform operations comprising:

generating a response to the voice input, the response including a list of information items to be presented to the user; and

outputting the information items via an auditory output mode, wherein in accordance with a determination that the electronic device is not in a vehicle, the information items are only presented on a display screen of the electronic device.

20. The non-transitory computer-readable medium of claim 19, the instructions further causing the processors to perform operations comprising, responsive to the determining that the electronic device is in a vehicle, limiting the amount of items in the list to a predetermined amount.

21. The non-transitory computer-readable medium of claim 17, wherein the voice input corresponds to content to be sent to a recipient, the instructions further causing the processors to perform operations comprising:

producing a message corresponding to the voice input, wherein the auditory output and the visual output include the message; and

requesting confirmation prior to sending the message to the recipient.

22. The non-transitory computer-readable medium of claim 21, wherein requesting confirmation comprises asking the user, via the auditory output mode, whether the text should be sent to the recipient.

23. The non-transitory computer-readable medium of claim 17, the instructions further causing the processors to perform operations comprising:

receiving a voice input comprising natural language information;

generating, based on the voice input, a plurality of candidate interpretations of user intent;

inferring a user intent based on the plurality of candidate interpretations of the user intent;

identifying a task based at least in part on the inferred user intent;

executing the identified task; and

in accordance with the execution of the identified task, generating a response to the user.

24. The non-transitory computer-readable storage medium of claim 17, wherein both the auditory output and the visual output paraphrase the voice input.

25. A system, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

automatically, without user input and without regard to whether a digital assistant application has been separately invoked by a user, determining that the electronic device is in a vehicle; and

responsive to the determining;

automatically invoking a listening mode of a virtual assistant implemented by the electronic device, wherein the listening mode causes the electronic device to automatically, without a physical input from the user, listen for voice input from the user for a predetermined time after the electronic device provides an auditory output;

receiving a voice input from the user; and

concurrently providing both an auditory output and a visual output on a display screen of the electronic device based on the voice input, wherein at least one of the auditory output and the visual output paraphrases the voice input.

26. The system of claim 25, wherein determining that the electronic device is in a vehicle comprises detecting that the electronic device is in communication with the vehicle.

27. The system of claim 25, the instructions further causing the processors to perform operations comprising:

generating a response to the voice input, the response including a list of information items to be presented to the user; and

outputting the information items via an auditory output mode, wherein in accordance with a determination that the electronic device is not in a vehicle, the information items are only presented on a display screen of the electronic device.

28. The system of claim 27, the instructions further causing the processors to perform operations comprising, responsive to the determining that the electronic device is in a vehicle, limiting the amount of items in the list to a predetermined amount.

29. The system of claim 25, wherein the voice input corresponds to content to be sent to a recipient, the instructions further causing the processors to perform operations comprising:

producing a message corresponding to the voice input, wherein the auditory output and the visual output include the message; and

requesting confirmation prior to sending the message to the recipient.

30. The system of claim 29, wherein requesting confirmation comprises asking the user, via the auditory output mode, whether the text should be sent to the recipient.

31. The system of claim 25, the instructions further causing the processors to perform operations comprising:

receiving a voice input comprising natural language information;

generating, based on the voice input, a plurality of candidate interpretations of user intent;

inferring a user intent based on the plurality of candidate interpretations of the user intent;

identifying a task based at least in part on the inferred user intent;

executing the identified task; and

in accordance with the execution of the identified task, generating a response to the user.

32. The system of claim 25, wherein both the auditory output and the visual output paraphrase the voice input.