Automatically adapting user interfaces for hands-free interaction

• US 10,496,753 B2
• Filed: 09/30/2011
• Issued: 12/03/2019
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gruber, Thomas Robert, Saddler, Harry J.
• Primary Examiner(s): Chuang, Jung-Mu T
• Application Number:
  US13/250,947
• Publication Number:
  US 20120022872A1
• Time in Patent Office: 2,986 Days
• Field of Search: 7042701, 704E11001
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A user interface for a system such as a virtual assistant is automatically adapted for hands-free use. A hands-free context is detected via automatic or manual means, and the system adapts various stages of a complex interactive system to modify the user experience to reflect the particular limitations of such a context. The system of the present invention thus allows for a single implementation of a complex system such as a virtual assistant to dynamically offer user interface elements and alter user interface behavior to allow hands-free use without compromising the user experience of the same system for hands-on use.

Claims

1. A computer-implemented method for adapting a user interface on a computing device having at least one processor, comprising:

performing, at the computing device, a plurality of steps including;

detecting whether a hands-free context is active;

prompting a user for an input;

receiving a user input comprising natural language information;

interpreting the received user input to derive a representation of a user intent, wherein the interpreting of the received user input comprises;

generating a plurality of candidate interpretations based on the received user input,

determining the representation of the user intent based on the plurality of candidate interpretations;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of the user intent;

executing the at least one task using the at least one parameter, to derive a result;

in accordance with the derived result,

paraphrasing at least a portion of the user input in a spoken form;

generating speech using a plurality of voices to differentiate paraphrased user input from other spoken output; and

providing an audio output of the generated speech;

wherein, responsive to a detection that the computing device is in the hands-free context, the user interface is adapted to display a subset of user-interaction mechanisms displayed with the hands-free context being inactive, the subset including at least one user-interaction mechanism.

2. The method of claim 1, wherein at least two interaction modes are available for user interaction with the computing device, and wherein:

responsive to the detection that the computing device is in the hands-free context, at least one of the steps of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and at least one parameter for the task, and generating the speech is performed using a first interaction mode adapted to hand s-free operation; and

responsive to the detection that the computing device is not in the hands-free context, at least one of the steps of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and at least one parameter for the task, and generating the speech is performed using a second interaction mode not adapted to hands-free operation.

3. The method of claim 1, wherein detecting whether a hands-free context is active comprises detecting a condition indicating a limitation in at least one of:

a ability of the user to view visual output presented by the computing device;

a ability of the user to interact with a graphical user interface presented by the computing device;

a ability of the user to use a physical component of the computing device;

a ability of the user to perform a touch input on the computing device;

a ability of the user to activate a switch on the computing device; and

a ability of the user to use a keyboard on the computing device.

4. The method of claim 1, wherein prompting the user for the input comprises:

responsive to the detection that the computing device is not in the hands-free context, prompting the user via a first output mode not adapted to the hands-free context; and

responsive to the detection that the computing device is in the hands-free context, prompting the user via a second output mode adapted to the hands-free context.

5. The method of claim 4, wherein:

prompting the user via the first output mode comprises prompting the user via a visual output mode; and

prompting the user via the second output mode comprises prompting the user via an auditory output mode.

6. The method of claim 5, wherein:

prompting the user via the visual output mode comprises displaying a prompt on a display screen; and

prompting the user via the auditory output mode comprises outputting a spoken prompt.

7. The method of claim 1, wherein interpreting the received user input comprises:

responsive to the detection that the computing device is in the hands-free context, interpreting the received user input using a vocabulary associated with hands-free operation.

8. The method of claim 1, wherein identifying at least one task and at least one parameter for the task comprises, responsive to the detection that the computing device is in the hands-free context, performing at least one task flow identification step associated with hands-free operation.

9. The method of claim 8, wherein performing the at least one task flow identification step associated with the hands-free operation comprises prompting the user to review and confirm entered content via a speech-based interface.

10. The method of claim 8, wherein performing the at least one task flow identification step associated with the hands-free operation comprises performing the at least one task flow step using an auditory output.

11. The method of claim 8, wherein performing the at least one task flow identification step associated with the hands-free operation comprises performing the at least one task flow step selected from a limited group of available task flow steps configured for the hands-free context.

12. The method of claim 1, wherein paraphrasing at least a portion of the user input in the spoken form comprises combining a dialog template with at least one item of personal data.

13. The method of claim 1, wherein detecting whether or not a hands-free context is active comprises at least one of:

receiving a user input specifying a hands-free context;

receiving data from at least one sensor indicating an environmental condition associated with a hands-free context;

detecting a connection of a peripheral device associated with a hands-free context;

detecting a disconnection of a peripheral device not associated with a hands-free context;

detecting communication with an onboard system of a vehicle;

detecting a current location; and

detecting a current speed.

14. The method of claim 1, wherein:

prompting the user for the input comprises prompting the user via a conversational interface; and

receiving the user input comprises receiving the user input via the conversational interface.

15. The method of claim 1, wherein receiving the user input comprises:

receiving a spoken input; and

converting the spoken input to a text representation.

16. The method of claim 1, wherein the computing device comprises at least one of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.

17. The method of claim 1, wherein the user interface is further adapted to display a visual output corresponding to the audio output of the speech.

18. A computer program product for interpreting a user input to perform a task on a computing device having at least one processor, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause the at least one processor of the computing device to perform a plurality of steps including;

detecting whether a hands-free context is active;

prompting a user for an input;

receiving the user input comprising natural language information;

interpreting the received user input to derive a representation of a user intent, wherein the interpreting of the received user input comprises;

generating a plurality of candidate interpretations based on the received user input,

determining the representation of the user intent based on the plurality of candidate interpretations;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of user intent;

executing the at least one task using the at least one parameter, to derive a result;

in accordance with the derived result,

paraphrasing at least a portion of the user input in a spoken form;

generating speech using a plurality of voices to differentiate paraphrased user input from other spoken output; and

providing an audio output of the generated speech;

wherein, responsive to a detection that the computing device is in the hands-free context, the computer program code is configured to cause the at least one processor to adapt the user interface to display a subset of user-interaction mechanisms displayed with the hands-free context being inactive, the subset including at least one user-interaction mechanism.

19. The computer program product of claim 18, wherein at least two interaction modes are available for user interaction with the computing device, and

wherein:

responsive to the detection that the computing device is in the hands-free context, the computer program code is configured to cause the at least one processor to perform at least one of the steps of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and at least one parameter for the task, and generating the speech using a first interaction mode adapted to hands-free operation; and

responsive to the detection that the computing device is not in the hands-free context, the computer program code is configured to cause the at least one processor to perform at least one of the steps of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and the at least one parameter for the task, and generating the speech using a second interaction mode not adapted to hands-free operation.

20. The computer program product of claim 18, wherein the computer program code configured to cause at least one processor to detect whether a hands-free context is active comprises the computer program code configured to cause the at least one processor to detect a condition indicating a limitation in at least one of:

a ability of the user to view visual output presented by the computing device;

a ability of the user to interact with a graphical user interface presented by the computing device;

a ability of the user to use a physical component of the computing device;

a ability of the user to perform a touch input on the computing device;

a ability of the user to activate a switch on the computing device; and

a ability of the user to use a keyboard on the computing device.

21. The computer program product of claim 18, wherein the computer program code configured to cause an output device to prompt the user for the input comprises:

the computer program code configured to cause the output device to, responsive to the detection that the computing device is not in the hands-free context, prompt the user via a first output mode not adapted to the hands-free context; and

the computer program code configured to cause the output device to, responsive to the detection that the computing device is in the hands-free context, prompt the user via a second output mode adapted to the hands-free context.

22. The computer program product of claim 18, wherein the computer program code configured to cause the at least one processor to interpret the received user input comprises:

the computer program code configured to cause the at least one processor to, responsive to the detection that the computing device is in the hands-free context, interpret the received user input using a vocabulary associated with hands-free operation.

23. The computer program product of claim 18, wherein the computer program code configured to cause the at least one processor to identify at least one task and at least one parameter for the task comprises the computer program code configured to cause the at least one processor to, responsive to the detection that the device is in the hands-free context, perform at least one task flow identification step associated with hands-free operation.

24. The computer program product of claim 18, wherein the computer program code configured to cause at least one processor to detect whether or not a hands-free context is active comprises the computer program code configured to cause the at least one processor to perform at least one of:

receiving a user input specifying a hands-free context;

receiving data from at least one sensor indicating an environmental condition associated with a hands-free context;

detecting a connection of a peripheral device associated with a hands-free context;

detecting a disconnection of a peripheral device not associated with a hands-free context;

detecting communication with an onboard system of a vehicle;

detecting a current location; and

detecting a current speed.

25. The computer program product of claim 18, wherein:

the computer program code configured to cause an output device to prompt the user for the input comprises the computer program code configured to cause the output device to prompt the user via a conversational interface; and

the computer program code configured to cause the at least one processor to receive the user input comprises the computer program code configured to cause the at least one processor to receive the user input via the conversational interface.

26. The computer program product of claim 18, wherein the computer program code configured to cause at least one processor to receive the user input comprises:

the computer program code configured to cause the at least one processor to receive a spoken input; and

the computer program code configured to cause the at least one processor to convert the spoken input to a text representation.

27. The computer program product of claim 18, wherein the computing device comprises at least one of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.

28. A system for interpreting a user input to perform a task on a computing device, comprising:

an output device, configured to prompt a user for an input;

an input device, configured to receive the user input comprising natural language information;

at least one processor, communicatively coupled to the output device and to the input device, configured to perform a plurality of steps including;

detecting whether a hands-free context is active;

interpreting the received user input to derive a representation of a user intent, wherein the interpreting of the received user input comprises;

generating a plurality of candidate interpretations based on the received user input,

determining the representation of the user intent based on the plurality of candidate interpretations;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of the user intent;

executing the at least one task using the at least one parameter, to derive a result;

in accordance with the derived result,

paraphrasing at least a portion of the user input in a spoken form;

generating speech using a plurality of voices to differentiate paraphrased user input from other spoken output;

wherein the output device is further configured to provide an audio output of the generated speech;

and wherein, responsive to a detection that the computing device is in the hands-free context, the user interface is adapted to display a subset of user-interaction mechanisms displayed with the hands-free context being inactive, the subset including at least one user-interaction mechanism.

29. The system of claim 28, wherein at least two interaction modes are available for user interaction, and wherein:

responsive to the detection that the computing device is in the hands-free context, at least one of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and at least one parameter for the task, and generating the speech is performed using a first interaction mode adapted to hands-free operation; and

responsive to the detection that the computing device is not in the hands-free context, at least one of prompting the user for the input, receiving the user input, interpreting the received user input, identifying the at least one task and at least one parameter for the task, and generating the speech is performed using a second interaction mode not adapted to hands-free operation.

30. The system of claim 28, wherein the at least one processor is configured to detect whether or not a hands-free context is active by detecting a condition indicating a limitation in at least one of:

a ability of the user to view visual output presented by the output device;

a ability of the user to interact with a graphical user interface presented by the output device;

a ability of the user to use a physical component of the computing device;

a ability of the user to perform a touch input on the input device;

a ability of the user to activate a switch on the computing device; and

a ability of the user to use a keyboard on the computing device.

31. The system of claim 28, wherein the input device is configured to prompt the user for input by:

responsive to the detection that the computing device is not in the hands-free context, prompting the user via a first output mode not adapted to the hands-free context; and

responsive to the detection that the computing device is in the hands-free context, prompting the user via a second output mode adapted to the hands-free context.

32. The system of claim 31, wherein:

the first output mode comprises a visual output mode; and

the second output mode comprises an auditory output mode;

and wherein the output device comprises;

a display screen, configured to prompt the user via the visual output mode; and

a speaker, configured to prompt the user via the auditory output mode.

33. The system of claim 32, wherein:

the display screen is configured to display a visual prompt; and

the speaker is configured to output a spoken prompt.

34. The system of claim 28, wherein, responsive to the detection that the computing device is in the hands-free context, the at least one processor is configured to interpret the received user input using a vocabulary associated with hands-free operation.

35. The system of claim 28, wherein, responsive to the detection that the computing device is in the hands-free context, the at least one processor is configured to identify the at least one task and the at least one parameter for the task by performing at least one task flow identification step associated with hands-free operation.

36. The system of claim 28, wherein the at least one processor is configure to detect whether or not the hands-free context is active based on at least one of:

a user input specifying a hands-free context;

data from at least one sensor indicating an environmental condition associated with a hands-free context;

a connection of a peripheral device associated with a hands-free context;

a disconnection of a peripheral device not associated with a hands-free context;

communication with an onboard system of a vehicle;

a current location; and

a current speed.

37. The system of claim 28, wherein:

the output device is configured to prompt the user for the input via a conversational interface; and

the input device is configured to receive the user input via the conversational interface.

38. The system of claim 28, wherein:

the input device comprises a microphone configured to receive a spoken input; and

the at least one processor is configured to convert the spoken input to a text representation.

39. The system of claim 28, wherein the computing device comprises at least one of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.