Dynamic device interaction adaptation based on user engagement

• US 10,732,826 B2
• Filed: 11/22/2017
• Issued: 08/04/2020
• Est. Priority Date: 11/22/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Klein, Christian, Glass, Andrew Stuart
• Primary Examiner(s): Rodriguez, Daniel
• Application Number:
  US15/821,488
• Publication Number:
  US 20190155495A1
• Time in Patent Office: 986 Days
• Field of Search: 715863
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

The techniques disclosed herein enable dynamic device interaction adaptation based on user engagement. In general, a computer can leverage sensors to determine a user'"'"'s level of engagement with one or more devices. As the user transitions his or her attention to different devices, the computer can utilize different interaction models to assist the user in interacting with each device that meets a threshold level of engagement with the user. Individual interaction models can configure the computer to direct input gestures received by the computing device to a selected device, and adapt the computer'"'"'s input modality to provide user interface controls that are optimized for the selected device. When a user is interacting with two or more devices, automatic transitions between different interaction models can help improve the user'"'"'s interaction with each device.

Claims

1. A computing device, comprising:

one or more processors;

a touchscreen for displaying graphical user interfaces and for receiving input gestures from a user to the computing device, wherein the computing device is a hand-held device;

a memory in communication with the one or more processors, the memory having computer-readable instructions stored thereupon which, when executed by the one or more processors, cause the computing device to;

receive image data from one or more sensors, wherein the image data indicates a gaze target of the user;

analyze the image data to determine a selected object based on the gaze target, wherein the selected object is the computing device when the gaze target is between a selection boundary and the computing device, and wherein the selected object is a remote device when the gaze target is between the selection boundary and the remote device;

in response to determining that the selected object is the computing device, selecting a first interaction model, wherein the first interaction model comprises displaying a first user interface of an application module on the touchscreen and causing the touchscreen to receive user gestures for providing input to the application module executing on the computing device; and

in response to determining that the selected object is a remote device in communication with the computing device, selecting a second interaction model, wherein the second interaction model causes the touchscreen to receive user gestures for providing input to the remote device, wherein the second interaction model causes a display of a second user interface that is configured and arranged according to functions of the remote device, wherein the selection boundary is adjusted in response to determining that the selected object is the computing device or the remote device.

2. The computing device of claim 1, wherein the instructions further cause the computing device to:

receive supplemental signals from one or more sensors, wherein the supplemental signals are generated by at least one of a location device, an audio generator, an infrared signal generator, or a sonar signal generator, wherein the supplemental signals indicate a location of the remote device;

analyze the supplemental signals to assist in the identification of the object in the gaze direction; and

determine that the object is the remote device when the supplemental signals indicate that an interaction of the user is focused on the remote device over the computing device.

3. The computing device of claim 1, wherein the remote device is a display screen in communication with the computing device, and wherein the second interaction model causes the computing device to display an indication of the input gestures received at the touchscreen on a graphical user interface rendered on the display screen.

4. The computing device of claim 1, wherein the remote device is a remote computer in communication with a display screen, and wherein the second interaction model causes the computing device to communicate the input gestures received at the touchscreen to the remote computer, the remote computer displaying an indication of the input gestures received at the touchscreen on a graphical user interface rendered on the display screen.

5. The computing device of claim 4, wherein the second interaction model causes the computing device to display a mirror of the graphical user interface rendered on the display screen, and wherein the second interaction model causes the computing device to display the indication of the input gestures on the touchscreen.

6. The computing device of claim 5, wherein the second interaction model causes the computing device to send instructions to the remote computer that cause the remote computer to display an indication of an input gesture of a second user received at the remote computer concurrently with the display of the indication of the input gestures received at the touchscreen.

7. The computing device of claim 1, wherein the instructions further cause the computing device to determine that the object is the remote device and that a second user is interacting with the remote device, selecting a third interaction model, wherein the third interaction model causes the touchscreen to receive user gestures for providing input to the remote device to influence an input of a second user providing input to the remote device.

8. The computing device of claim 1, wherein the remote device is an embedded device, and wherein the second interaction model causes the computing device to communicate the input gestures received at the touchscreen to the embedded device, the input gestures causing the embedded device to perform a function of the embedded device in response to the input gestures.

9. The computing device of claim 8, wherein the second interaction model causes the computing device to display graphical user interface controls that enable the user to provide the input gestures for invoking the function of the embedded device.

10. The computing device of claim 1, wherein the instructions further cause the computing device to:

in response to determining that the object is an unidentified object, selecting a default interaction model, wherein the default interaction model causes the computing device to display graphical user interface controls that enable the user to provide the input gestures for invoking a default function, and

processing the input gestures for invoking the default function.

11. The computing device of claim 10, wherein the default function comprises sending commands to a remote service, wherein processing the input gestures for invoking the default function comprises transmitting commands based on the input gestures to the remote service.

12. The computing device of claim 1, wherein the instructions further cause the computing device to:

generate supplemental signal data defining patterns of user engagement with the computing device and the remote device over time, wherein analyzing the image data to identify the object in the gaze direction further comprises analysis of the supplemental signal data defining patterns of user engagement with the computing device and the remote device over time.

13. The computing device of claim 1, wherein the selection boundary is at a first distance from the computing device and the selection boundary is at a second distance from the remote device, and wherein the instructions further cause the computing device to:

adjust the selection boundary by causing the first distance (D1) to be shorter than the second distance (D2) when the first interaction model is selected; and

adjust the selection boundary by causing the second distance (D2) to be shorter than the first distance (D1) when the second interaction model is selected.

14. A method, comprising:

receiving image data, at a computing device, from one or more sensors, wherein the image data indicates a gaze target of a user, the computing device comprising a touchscreen for displaying graphical user interfaces and for receiving input gestures from a user to the computing device, wherein the computing device is a hand-held device;

analyzing the image data to select the computing device or a remote device in communication with the computing device, wherein the computing device is selected when the gaze target is directed to the computing device, and wherein the remote device is selected when the gaze direction is directed to the remote device, and wherein the selected object is the computing device when the gaze target is between a selection boundary and the computing device, and wherein the selected object is a remote device when the gaze target is between the selection boundary and the remote device;

in response to determining that the selected gaze direction is directed to the computing device, selecting the computing device and a first interaction model, wherein the first interaction model comprises displaying a first user interface of an application module on the touchscreen and causing the touchscreen to receive user gestures for providing input to the application module executing on the computing device; and

in response to determining that the selected gaze direction is directed to the remote device, selecting the remote device and a second interaction model, wherein the second interaction model causes the touchscreen to receive user gestures for providing input to the remote device, wherein the second interaction model causes a display of a second user interface that is configured and arranged according to functions of the remote device, wherein the selection boundary is adjusted in response to determining that the selected object is the computing device or the remote device.

15. The method of claim 14, further comprising:

receiving supplemental signals from one or more sensors, wherein the supplemental signals are generated by at least one of a location device, an audio generator, an infrared signal generator, or a sonar signal generator, wherein the supplemental signals indicate a location of the remote device;

analyzing the supplemental signals to assist in the selection of the computing device or the remote device;

selecting the remote device when the supplemental signals indicate that an interaction of the user is focused on the remote device over the computing device; and

selecting the computing device when the supplemental signals indicate that the interaction of the user is focused on the computing device over the remote device.

16. The method of claim 14, wherein the remote device is a display screen in communication with the computing device, and wherein the second interaction model causes the computing device to display an indication of the input gestures received at the touchscreen on a graphical user interface rendered on the display screen.

17. The method of claim 14, wherein the remote device is a remote computer in communication with a display screen, and wherein the second interaction model causes the computing device to communicate the input gestures received at the touchscreen to the remote computer, the remote computer displaying an indication of the input gestures received at the touchscreen on a graphical user interface rendered on the display screen.

18. The method of claim 17, wherein the second interaction model causes the computing device to display a mirror of the graphical user interface rendered on the display screen, and wherein the second interaction model causes the computing device to display the indication of the input gestures on the touchscreen.

19. A non-transitory computer-readable medium comprising instructions which, when executed by one or more processors of a computing device comprising a touchscreen for displaying graphical user interfaces and for receiving input gestures from a user, cause the computing device to:

receive image data from one or more sensors, wherein the image data indicates a gaze target of the user;

analyze the image data to determine a selected an object based on the gaze target, wherein the selected object is the computing device when the gaze target is between a selection boundary and the computing device, and wherein the selected object is a remote device when the gaze target is between the selection boundary and the remote device;

in response to determining that the selected object is the computing device, selecting a first interaction model, wherein the first interaction model comprises displaying a first user interface of an application module on the touchscreen and causing the touchscreen to receive user gestures for providing input to the application module executing on the computing device; and

in response to determining that the selected object is a remote device in communication with the computing device, selecting a second interaction model, wherein the second interaction model causes the touchscreen to receive user gestures for providing input to the remote device, wherein the second interaction model causes a display of a second user interface that is configured and arranged according to functions of the remote device, wherein the selection boundary is adjusted in response to determining that the selected object is the computing device or the remote device, wherein the computing device is a hand-held device.