Contextual user interface based on environment

• US 10,359,993 B2
• Filed: 05/04/2017
• Issued: 07/23/2019
• Est. Priority Date: 01/20/2017
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Roman, Manuel, Segal, Mara Clair, Desai, Dwipal, Rubin, Andrew E.
• Primary Examiner(s): Pitaro, Ryan F
• Application Number:
  US15/587,201
• Publication Number:
  US 20180210700A1
• Time in Patent Office: 810 Days
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2221/2111 Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

A contextual user interface based on environment is described. An assistant device can determine that speech has been spoken and determine the context of an environment of that speech. A user interface can then be generated based on the context of the environment and the content of the speech. Different context can result in different user interfaces being generated.

Claims

1. A home assistant device, comprising:

a display screen;

a microphone;

one or more processors; and

memory storing instructions, wherein the one or more processors are configured to execute the instructions such that the one or more processors and memory are configured to;

determine that first speech has been spoken in an environment of the home assistant device using the microphone;

determine a first context of the environment of the home assistant device, the first context of the environment including characteristics of the first speech and one or more of a location of a user providing the first speech, a time of the first speech, a user identity corresponding to the user providing the first speech, a skill level with interacting with the home assistant device of the user providing the first speech, or a schedule of the user providing the first speech;

determine a first distance from a source of the first speech to the home assistant device;

display a first graphical user interface (GUI) for the home assistant device on the display screen to provide a response regarding the first speech, the first GUI based on the first context of the environment including the characteristics of the first speech, the first distance, and content of the first speech;

determine that second speech has been spoken in the environment of the home assistant device using the microphone, the first speech and the second speech including the same content;

determine a second context of the environment of the home assistant device, the second context of the environment including characteristics of the second speech and one or more of a location of a user providing the second speech, a time of the second speech, a user identity corresponding to the user providing the second speech, a skill level with interacting with the home assistant device of the user providing the second speech, a schedule of the user providing the second speech, the first context and the second context being different, wherein the characteristics of the first speech is different than the characteristics of the second speech; and

determine a second distance from a source of the second speech to the home assistant device, the second distance being farther than the first distance;

display a second GUI for the home assistant device on the display screen to provide a response regarding the second speech, the second GUI based on the second context of the environment including the characteristics of the second speech, the second distance, and content of the second speech, the first GUI and the second GUI providing different content, the second GUI including content also included in the first GUI, the content in the second GUI being a different size than the content in the first GUI based on the second distance being farther than the first distance, wherein the content on the second GUI is updated at a different speed than the content on the first GUI based on the first context and the second context being different.

2. A method for providing a contextual user interface on an assistant device, comprising:

determining, by a processor, that a first speech has been spoken;

determining, by the processor, a first context of an environment corresponding to the first speech, the first context of the environment including how the first speech was spoken;

determining a first distance from a source of the first speech to the assistant device;

providing, by the processor, a first user interface based on the first context of the environment including the how the first speech was spoken, the first distance, and content of the first speech;

determining, by the processor, that a second speech has been spoken, the second speech spoken at a different time than the first speech;

determining, by the processor, a second context of the environment corresponding to the second speech, the second context of the environment including how the second speech was spoken, the first context and the second context being different, wherein how the first speech was spoken is different than how the second speech was spoken;

determining a second distance from a source of the second speech to the assistant device, the second distance being farther than the first distance; and

providing, by the processor, a second user interface based on the second context of the environment including how the second speech was spoken, the second distance, and content of the second speech, the content of the first speech and the second speech being similar, the first user interface and the second user interface being different, the second user interface including content also included in the first user interface, the content in the second user interface being a different size than the content in the first user interface based on the second distance being farther than the first distance, wherein the content on the second user interface is updated at a different speed than the content on the first user interface based on the first context and the second context being different.

3. The method of claim 2, wherein the first context is based on one or both of audio or visual determinations of a surrounding environment of the assistant device that the first speech and the second speech is directed.

4. The method of claim 2, wherein the first context includes a first interaction corresponding to the first speech at the first distance, wherein the second context includes a second interaction corresponding to the second speech at the second distance, the first distance and the second distance being different.

5. The method of claim 2, wherein the first context includes a first user providing the first speech, the second context includes a second user providing the second speech, the first user and the second user being different.

6. The method of claim 5, wherein the first user is associated with a first skill level with interacting with the assistant device, the second user is associated with a second skill level with interacting with the assistant device, the first skill level and the second skill level being different, the first context based on the first skill level, and the second context based on the second skill level.

7. The method of claim 2, wherein the first context and the second context include one or more of a user interacting with an assistant device, people in the environment around the assistant device, a time of an interaction with the assistant device, a location of the user interacting with the assistant device, or a skill level of the user interacting with the assistant device.

8. The method of claim 2, further comprising:

determining, by the processor, a change in the environment; and

generating, by the processor, a third user interface based on one or more of the first context or the second context in response to the change in the environment to maintain privacy expectations of one or more users present in the environment.

9. An electronic device, comprising:

one or more processors; and

memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and memory are configured to;

determine that a first speech has been spoken;

determine a first context of an environment corresponding to the first speech, the first context of the environment including a volume of the first speech;

determine a first distance from a source of the first speech to an assistant device;

generate a first user interface based on the first context of the environment including the volume of the first speech, the first distance, and content of the first speech;

determine that a second speech has been spoken, the second speech spoken at a different time than the first speech;

determine a second context of the environment corresponding to the second speech, the second context of the environment including a volume of the second speech, the first context and the second context being different, wherein the volume of the first speech is different than the volume of the second speech;

determine a second distance from a source of the second speech to the assistant device, the second distance being farther than the first distance; and

generate a second user interface based on the second context of the environment including the volume of the second speech, the second distance, and content of the second speech, the content of the first speech and the second speech being similar, the first user interface and the second user interface being different, the second user interface including content also included in the first user interface, the content in the second user interface being a different size than the content in the first user interface based on the second distance being farther than the first distance, wherein the content on the second user interface is updated at a different speed than the content on the first user interface based on the first context and the second context being different.

10. The electronic device of claim 9, wherein the first context is based on one or both of audio or visual determinations of a surrounding environment of the assistant device that the first speech and the second speech is directed.

11. The electronic device of claim 9, wherein the first context includes a first interaction corresponding to the first speech at the first distance, wherein the second context includes a second interaction corresponding to the second speech at the second distance, the first distance and the second distance being different.

12. The electronic device of claim 9, wherein the first context includes a first user providing the first speech, the second context includes a second user providing the second speech, the first user and the second user being different.

13. The electronic device of claim 12, wherein the first user is associated with a first skill level with interacting with the assistant device, the second user is associated with a second skill level with interacting with the assistant device, the first skill level and the second skill level being different, the first context based on the first skill level, and the second context based on the second skill level.

14. The electronic device of claim 9, wherein the first context and the second context include one or more of a user interacting with an assistant device, people in the environment around the assistant device, a time of an interaction with the assistant device, a location of the user interacting with the assistant device, or a skill level of the user interacting with the assistant device.

15. The electronic device of claim 9, wherein the processor is configured to execute the instructions such that the processor and memory are configured to:

determine a change in the environment; and

generate a third user interface based on one or more of the first context or the second context in response to the change in the environment to maintain privacy expectations of one or more users present in the environment.

16. A computer program product, comprising one or more non-transitory computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:

determine that a first speech has been spoken;

determine a first context of an environment corresponding to the first speech, the first context of the environment including an indication that the first speech was spoken at a first speed;

determine a first distance from a source of the first speech to an assistant device;

generate a first user interface based on the first context of the environment including the indication that the first speech was spoken at the first speed, the first distance, and content of the first speech;

determine that a second speech has been spoken, the second speech spoken at a different time than the first speech;

determine a second context of the environment corresponding to the second speech, the second context of the environment including an indication that the second speech was spoken at a second speed, the first context and the second context being different, wherein the first speed and the second speed are different;

determine a second distance from a source of the second speech to the assistant device, the second distance being farther than the first distance; and

generate a second user interface based on the second context of the environment including the second speed of the second speech, the second distance, and content of the second speech, the content of the first speech and the second speech being similar, the first user interface and the second user interface being different, the second user interface including content also included in the first user interface, the content in the second user interface being a different size than the content in the first user interface based on the second distance being farther than the first distance, wherein the content on the second user interface is updated at a different speed than the content on the first user interface based on the first context and the second context being different.

17. The computer program product of claim 16, wherein the first context is based on one or both of audio or visual determinations of a surrounding environment of the assistant device that the first speech and the second speech is directed.

18. The computer program product of claim 16, wherein the first context includes a first interaction corresponding to the first speech at the first distance, wherein the second context includes a second interaction corresponding to the second speech at the second distance, the first distance and the second distance being different.

19. The computer program product of claim 16, wherein the first context includes a first user providing the first speech, the second context includes a second user providing the second speech, the first user and the second user being different.

20. The computer program product of claim 19, wherein the first user is associated with a first skill level with interacting with the assistant device, the second user is associated with a second skill level with interacting with the assistant device, the first skill level and the second skill level being different, the first context based on the first skill level, and the second context based on the second skill level.

21. The computer program product of claim 16, wherein the first context and the second context include one or more of a user interacting with an assistant device, people in the environment around the assistant device, a time of an interaction with the assistant device, a location of the user interacting with the assistant device, or a skill level of the user interacting with the assistant device.

22. The computer program product of claim 16, wherein the computer program instructions cause the one or more computing devices to:

determine a change in the environment; and

generate a third user interface based on one or more of the first context or the second context in response to the change in the environment to maintain privacy expectations of one or more users present in the environment.

23. The computer program product of claim 16, wherein a location of the content on the second user interface is different than a location of the content on the first user interface based on the second distance being farther than the first distance.