Automatically Adapting User Interfaces For Hands-Free Interaction

• US 20120022872A1
• Filed: 09/30/2011
• Published: 01/26/2012
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gruber, Thomas Robert, Saddler, Harry J.
• Granted Patent: US 10,496,753 B2
• Time in Patent Office: Days
• US Class Current: 704/270.1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A user interface for a system such as a virtual assistant is automatically adapted for hands-free use. A hands-free context is detected via automatic or manual means, and the system adapts various stages of a complex interactive system to modify the user experience to reflect the particular limitations of such a context. The system of the present invention thus allows for a single implementation of a complex system such as a virtual assistant to dynamically offer user interface elements and alter user interface behavior to allow hands-free use without compromising the user experience of the same system for hands-on use.

Claims

1. A computer-implemented method for interpreting user input to perform a task on a computing device having at least one processor, comprising:

at a processor, detecting whether or not a hands-free context is active;

at an output device, prompting a user for input;

at an input device, receiving user input;

at the processor, interpreting the received user input to derive a representation of user intent;

at the processor, identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of user intent;

at the processor, executing the at least one task using the at least one parameter, to derive a result;

at the processor, generating a dialog response based on the derived result; and

at the output device, outputting the generated dialog response;

wherein, responsive to detection that the device is in a hands-free context, at least one of the steps of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed in a manner consistent with limitations associated with the hands-free context.

2. The method of claim 1, wherein at least two interaction modes are available for user interaction with the computing device, and wherein:

responsive to detection that the device is in a hands-free context, at least one of the steps of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed using a first interaction mode adapted to hands-free operation; and

responsive to detection that the device is not in a hands-free context, at least one of the steps of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed using a second interaction mode not adapted to hands-free operation.

3. The method of claim 1, wherein detecting whether or not a hands-free context is active comprises detecting a condition indicating a limitation in at least one selected from the group consisting of:

the ability of a user to view visual output presented by the computing device;

the ability of a user to interact with a graphical user interface presented by the computing device;

the ability of a user to use a physical component of the computing device;

the ability of a user to perform touch input on the computing device;

the ability of a user to activate a switch on the computing device; and

the ability of a user to use a keyboard on the computing device.

4. The method of claim 1, wherein prompting a user for input comprises:

responsive to detection that the device is not in a hands-free context, prompting the user via a first output mode not adapted to the hands-free context; and

responsive to detection that the device is in a hands-free context, prompting the user via a second output mode adapted to the hands-free context.

5. The method of claim 4, wherein:

prompting the user via a first output mode comprises prompting the user via a visual output mode; and

prompting the user via a second output mode comprises prompting the user via an auditory output mode.

6. The method of claim 5, wherein:

prompting the user via a visual output mode comprises displaying a prompt on a display screen; and

prompting the user via an auditory output mode comprises outputting a spoken prompt.

7. The method of claim 1, wherein interpreting the received user input comprises:

responsive to detection that the device is in a hands-free context, interpreting the received user input using a vocabulary associated with hands-free operation.

8. The method of claim 1, wherein identifying at least one task and at least one parameter for the task comprises, responsive to detection that the device is in a hands-free context, performing at least one task flow identification step associated with hands-free operation.

9. The method of claim 8, wherein performing at least one task flow identification step associated with hands-free operation comprises prompting the user to review and confirm entered content via a speech-based interface.

10. The method of claim 8, wherein performing at least one task flow identification step associated with hands-free operation comprises performing at least one task flow step using auditory output.

11. The method of claim 8, wherein performing at least one task flow identification step associated with hands-free operation comprises performing at least one task flow step selected from a limited group of available task flow steps suited to the hands-free context.

12. The method of claim 1, wherein generating a dialog response comprises, responsive to detection that the device is in a hands-free context, generating a dialog response in a speech-based output mode.

13. The method of claim 12, wherein generating a dialog response in a speech-based output mode comprises paraphrasing at least a portion of the user input in spoken form.

14. The method of claim 13, wherein generating a dialog response in a speech-based output mode further comprises generating speech using a plurality of voices to differentiate paraphrased user input from other spoken output.

15. The method of claim 12, wherein generating a dialog response in a speech-based output mode comprises combining a dialog template with at least one item of personal data.

16. The method of claim 1, wherein detecting whether or not a hands-free context is active comprises at least one selected from the group consisting of:

receiving user input specifying a hands-free context;

receiving data from at least one sensor indicating an environmental condition associated with a hands-free context;

detecting connection of a peripheral device associated with a hands-free context;

detecting disconnection of a peripheral device not associated with a hands-free context;

detecting communication with an onboard system of a vehicle;

detecting current location; and

detecting current speed.

17. The method of claim 1, wherein:

prompting the user comprises prompting the user via a conversational interface; and

receiving user input comprises receiving user input via the conversational interface.

18. The method of claim 1, wherein receiving user input comprises:

receiving spoken input; and

converting the spoken input to a text representation.

19. The method of claim 1, wherein the computing device comprises at least one selected from the group consisting of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.

20. A computer program product for interpreting user input to perform a task on a computing device having at least one processor, comprising:

a nontransitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

detecting whether or not a hands-free context is active;

causing an output device to prompt a user for input;

receiving user input via an input device;

interpreting the received user input to derive a representation of user intent;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of user intent;

executing the at least one task using the at least one parameter, to derive a result;

generating a dialog response based on the derived result; and

causing the output device to output the generated dialog response;

wherein, responsive to detection that the device is in a hands-free context, the computer program code is configured to cause at least one processor to perform at least one of the steps of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response in a manner consistent with limitations associated with the hands-free context.

21. The computer program product of claim 20, wherein at least two interaction modes are available for user interaction with the computing device, and wherein:

responsive to detection that the device is in a hands-free context, the computer program code is configured to cause at least one processor to perform at least one of the steps of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response using a first interaction mode adapted to hands-free operation; and

responsive to detection that the device is not in a hands-free context, the computer program code is configured to cause at least one processor to perform at least one of the steps of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response using a second interaction mode not adapted to hands-free operation.

22. The computer program product of claim 20, wherein the computer program code configured to cause at least one processor to detect whether or not a hands-free context is active comprises computer program code configured to cause at least one processor to detect a condition indicating a limitation in at least one selected from the group consisting of:

the ability of a user to view visual output presented by the computing device;

the ability of a user to interact with a graphical user interface presented by the computing device;

the ability of a user to use a physical component of the computing device;

the ability of a user to perform touch input on the computing device;

the ability of a user to activate a switch on the computing device; and

the ability of a user to use a keyboard on the computing device.

23. The computer program product of claim 20, wherein the computer program code configured to cause an output device to prompt a user for input comprises:

computer program code configured to cause an output device to, responsive to detection that the device is not in a hands-free context, prompt the user via a first output mode not adapted to the hands-free context; and

computer program code configured to cause an output device to, responsive to detection that the device is in a hands-free context, prompt the user via a second output mode adapted to the hands-free context.

24. The computer program product of claim 20, wherein the computer program code configured to cause at least one processor to interpret the received user input comprises:

computer program code configured to cause at least one processor to, responsive to detection that the device is in a hands-free context, interpret the received user input using a vocabulary associated with hands-free operation.

25. The computer program product of claim 20, wherein the computer program code configured to cause at least one processor to identify at least one task and at least one parameter for the task comprises computer program code configured to cause at least one processor to, responsive to detection that the device is in a hands-free context, perform at least one task flow identification step associated with hands-free operation.

26. The computer program product of claim 20, wherein the computer program code configured to cause at least one processor to generate a dialog response comprises computer program code configured to cause at least one processor to, responsive to detection that the device is in a hands-free context, generate a dialog response in a speech-based output mode.

27. The computer program product of claim 20, wherein the computer program code configured to cause at least one processor to detect whether or not a hands-free context is active comprises computer program code configured to cause at least one processor to perform at least one selected from the group consisting of:

receiving user input specifying a hands-free context;

receiving data from at least one sensor indicating an environmental condition associated with a hands-free context;

detecting connection of a peripheral device associated with a hands-free context;

detecting disconnection of a peripheral device not associated with a hands-free context;

detecting communication with an onboard system of a vehicle;

detecting current location; and

detecting current speed.

28. The computer program product of claim 20, wherein:

the computer program code configured to cause an output device to prompt the user comprises computer program code configured to cause an output device to prompt the user via a conversational interface; and

the computer program code configured to cause at least one processor to receive user input comprises computer program code configured to cause at least one processor to receive user input via the conversational interface.

29. The computer program product of claim 20, wherein the computer program code configured to cause at least one processor to receive user input comprises:

computer program code configured to cause at least one processor to receive spoken input; and

the computer program code configured to cause at least one processor to convert the spoken input to a text representation.

30. The computer program product of claim 20, wherein the computing device comprises at least one selected from the group consisting of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.

31. A system for interpreting user input to perform a task on a computing device, comprising:

an output device, configured to prompt a user for input;

an input device, configured to receive user input;

at least one processor, communicatively coupled to the output device and to the input device, configured to perform the steps of;

detecting whether or not a hands-free context is active;

interpreting the received user input to derive a representation of user intent;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of user intent;

executing the at least one task using the at least one parameter, to derive a result; and

generating a dialog response based on the derived result; and

wherein the output device is further configured to output the generated dialog response;

and wherein, responsive to detection that the device is in a hands-free context, at least one of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed in a manner consistent with limitations associated with the hands-free context.

32. The system of claim 31, wherein at least two interaction modes are available for user interaction, and wherein:

responsive to detection that the device is in a hands-free context, at least one of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed using a first interaction mode adapted to hands-free operation; and

responsive to detection that the device is not in a hands-free context, at least one of prompting the user for input, receiving user input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed using a second interaction mode not adapted to hands-free operation.

33. The system of claim 31, wherein the at least one processor is configured to detect whether or not a hands-free context is active by detecting a condition indicating a limitation in at least one selected from the group consisting of:

the ability of a user to view visual output presented by the output device;

the ability of a user to interact with a graphical user interface presented by the output device;

the ability of a user to use a physical component of the computing device;

the ability of a user to perform touch input on the input device;

the ability of a user to activate a switch on the computing device; and

the ability of a user to use a keyboard on the computing device.

34. The system of claim 31, wherein the input device is configured to prompt a user for input by:

responsive to detection that the device is not in a hands-free context, prompting the user via a first output mode not adapted to the hands-free context; and

responsive to detection that the device is in a hands-free context, prompting the user via a second output mode adapted to the hands-free context.

35. The system of claim 34, wherein:

the first output mode comprises a visual output mode; and

the second output mode comprises an auditory output mode;

and wherein the output device comprises;

a display screen, configured to prompt the user via the visual output mode; and

a speaker, configured to prompt the user via the auditory output mode.

36. The system of claim 35, wherein:

the display screen is configured to display a visual prompt; and

the speaker is configured to output a spoken prompt.

37. The system of claim 31, wherein, responsive to detection that the device is in a hands-free context, the at least one processor is configured to interpret the received user input using a vocabulary associated with hands-free operation.

38. The system of claim 31, wherein, responsive to detection that the device is in a hands-free context, the at least one processor is configured to identify at least one task and at least one parameter for the task by performing at least one task flow identification step associated with hands-free operation.

39. The system of claim 31, wherein, responsive to detection that the device is in a hands-free context, the at least one processor is configured to generate a dialog response in a speech-based output mode.

40. The system of claim 31, wherein the at least one processor is configure to detect whether or not a hands-free context is active based on at least one selected from the group consisting of:

user input specifying a hands-free context;

data from at least one sensor indicating an environmental condition associated with a hands-free context;

connection of a peripheral device associated with a hands-free context;

disconnection of a peripheral device not associated with a hands-free context;

communication with an onboard system of a vehicle;

current location; and

current speed.

41. The system of claim 31, wherein:

the output device is configured to prompt the user via a conversational interface; and

the input device is configured to receive user input via the conversational interface.

42. The system of claim 31, wherein:

the input device comprises a microphone configured to receive spoken input; and

the at least one processor is configured to convert the spoken input to a text representation.

43. The system of claim 31, wherein the computing device comprises at least one selected from the group consisting of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.