Detecting user focus on hinged multi-screen device

• US 10,331,190 B2
• Filed: 11/09/2016
• Issued: 06/25/2019
• Est. Priority Date: 11/09/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Veiga, Alexandre da, Sylvan, Roger Sebastian, Sipko, Jeffrey
• Primary Examiner(s): Pervan, Michael
• Application Number:
  US15/347,524
• Publication Number:
  US 20180129262A1
• Time in Patent Office: 958 Days
• Field of Search: 345 11, 345 13
• CPC Class Codes: G06F 1/1618 the display being foldable up to the back of the other housing with a single degree of freedom, e.g. by 360° rotation over the axis defined by the rear edge of the base enclosure
• CPC Class Codes: G06F 1/162 changing, e.g. reversing, the face orientation of the screen with a two degrees of freedom mechanism, e.g. for folding into tablet PC like position or orienting towards the direction opposite to the user to show to a second user
• CPC Class Codes: G06F 1/1647 including at least an additional display G06F1/1692 takes precedence
• CPC Class Codes: G06F 1/1677 for detecting open or closed state or particular intermediate positions assumed by movable parts of the enclosure, e.g. detection of display lid position with respect to main body in a laptop, detection of opening of the cover of battery compartment
• CPC Class Codes: G06F 1/1681 Details related solely to hinges hinge details related to the transmission of signals or power are classified in G06F1/1683
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G09G 2330/021 Power management, e.g. power saving
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A mobile computing device is provided that includes a processor, an accelerometer, two or more display devices, and a housing including the processor, the accelerometer, and the two or more display devices, determine a current user focus indicating that a first display device of the pair of display devices is being viewed by the user, and that a second display device of the pair of display devices is not being viewed by the user, detect a signature gesture input based on accelerometer data received via the accelerometer detecting that the mobile computing device has been rotated more than a threshold degree, determine that the current user focus has changed from the first display device to the second display device based on at least detecting the signature gesture input, and perform a predetermined action based on the current user focus.

Claims

1. A mobile computing device comprising:

a processor;

an accelerometer;

two or more display devices; and

a housing including the processor, the accelerometer, and the two or more display devices, the housing including a hinge between a pair of display devices of the two or more display devices, the hinge being configured to permit the pair of display devices to rotate between angular orientations from a face-to-face angular orientation to a back-to-back angular orientation;

the processor being configured to;

detect a current angular orientation between the pair of display devices indicating that the pair of display devices are facing away from each other;

detect a first touch input via a capacitive touch sensor of a first display device of the pair of display devices, a second touch input via a capacitive touch sensor of a second display device of the pair of display devices, and a third touch input via a capacitive touch sensor of a side of the mobile computing device;

determine that the first touch input is an intended touch input and the second touch input is an incidental touch input;

determine an orientation of the mobile computing device based on at least the first touch input, the second touch input, and the third touch input; and

determine which display device of the two or more display devices has a current user focus based on at least the determined orientation of the mobile computing device; and

perform a predetermined action based on the current user focus.

2. The mobile computing device of claim 1, wherein to perform the predetermined action, the processor is further configured to reduce a power usage of a display device that does not have the current user focus.

3. The mobile computing device of claim 1, the processor being further configured to:

display content on the first display device that has the current user focus;

detect a signature gesture input based on accelerometer data received via the accelerometer detecting that the mobile computing device has been rotated more than a threshold degree;

determine that the current user focus has changed from the first display device to the second display device based on at least detecting the signature gesture input; and

wherein to perform the predetermined action, the processor is further configured to, based on at least determining that the current user focus has changed from the first display device to the second display device, switch display of the content from the first display device to the second display device.

4. The mobile computing device of claim 3, wherein to detect the signature gesture input, the processor is further configured to:

detect a rotation axis around which the mobile computing device has been rotated more than the threshold degree based on accelerometer data received via the accelerometer; and

detect the signature gesture input based on at least detecting a longitudinal or lateral rotation axis.

5. The mobile computing device of claim 1, wherein the processor is further configured to execute one or more application programs;

and

wherein to perform the predetermined action, the processor is further configured to send the current user focus to the one or more application programs.

6. The mobile computing device of claim 1, wherein to perform the predetermined action, the processor is further configured to suppress touch input received by a capacitive touch sensor of a display device that does not have the current user focus.

7. The mobile computing device of claim 1, wherein to determine the current user focus, the processor is further configured to:

detect a first image of a surrounding in front of the first display device of the pair of display devices via a first camera and a second image of a surrounding in front of the second display device of the pair of display devices via a second camera;

process the first image and the second image to detect facial features;

based on at least detecting facial features in the first image, determine that the first display device has the current user focus indicating that the first display device of the pair of display devices is being viewed by the user; and

based on at least detecting no facial features in the second image, determine that the second display device does not have the current user focus indicating that the second display device of the pair of display devices is not being viewed by the user.

8. A mobile computing device comprising:

a processor;

an accelerometer;

two or more display devices; and

a housing including the processor, the accelerometer, and the two or more display devices, the housing including a hinge between a pair of display devices of the two or more display devices, the hinge being configured to permit the pair of display devices to rotate between angular orientations from a face-to-face angular orientation to a back-to-back angular orientation;

the processor being configured to;

detect a current angular orientation between the pair of display devices indicating that the pair of display devices are facing away from each other;

detect a first ambient light level for a surrounding in front of a first display device of the pair of display devices via a first ambient light sensor and a second ambient light level for a surrounding in front of a second display device of the pair of display devices via a second ambient light sensor;

determine that the first ambient light level is above a threshold light level and that the second ambient light level is below the threshold light level;

based on at least determining that the first ambient light level is above the threshold light level, determine that the first display device has the current user focus indicating that the first display device of the pair of display devices is being viewed by the user; and

based on at least determining that the second ambient light level is below the threshold light level, determine that the second display device does not have the current user focus indicating that the second display device of the pair of display devices is not being viewed by the user; and

perform a predetermined action based on the current user focus.

9. A mobile computing device comprising:

a processor;

an accelerometer;

two or more display devices; and

a housing including the processor, the accelerometer, and the two or more display devices, the housing including a hinge between a pair of display devices of the two or more display devices, the hinge being configured to permit the pair of display devices to rotate between angular orientations from a face-to-face angular orientation to a back-to-back angular orientation;

the processor being configured to;

detect a current angular orientation between the pair of display devices indicating that the pair of display devices are facing away from each other;

detect a first depth image of a surrounding in front of a first display device of the pair of display devices via a first depth camera and a second depth image of a surrounding in front of a second display device of the pair of display devices via a second depth camera;

determine that the first depth image includes depth values within a depth threshold and that the second depth image does not include depth values with the depth threshold;

based on at least determining that the first depth image includes depth values within the depth threshold, determine that the first display device has a current user focus indicating that the first display device of the pair of display devices is being viewed by the user; and

based on at least determining that the second depth image does not include depth values within the depth threshold, determine that the second display device does not have the current user focus indicating that the second display device of the pair of display devices is not being viewed by the user; and

perform a predetermined action based on the current user focus.

10. A computer-implemented method comprising:

detecting a current angular orientation between a pair of display devices of a mobile computing device, indicating that the pair of display devices are facing away from each other;

detecting a first touch input via a capacitive touch sensor of a first display device of the pair of display devices, a second touch input via a capacitive touch sensor of a second display device of the pair of display devices, and a third touch input via a capacitive touch sensor of a side of the mobile computing device;

determining that the first touch input is an intended touch input and the second touch input is an incidental touch input;

determining an orientation of the mobile computing device based on at least the first touch input, the second touch input, and the third touch input; and

determining which display device of the two or more display devices has a current user focus based on at least the determined orientation of the mobile computing device; and

performing a predetermined action based on the current user focus.

11. The computer-implemented method of claim 10, wherein performing the predetermined action include reducing a power usage of a display device that does not have the current user focus.

12. The computer-implemented method of claim 10, further comprising:

displaying content on the first display device that has the current user focus;

detecting a signature gesture input based on accelerometer data received via an accelerometer detecting that a mobile computing device including the pair of display devices has been rotated more than a threshold degree;

determining that the current user focus has changed from the first display device to the second display device based on at least detecting the signature gesture input; and

wherein performing the predetermined action includes, based on at least determining that the current user focus has changed from the first display device to the second display device, switching display of the content from the first display device to the second display device.

13. The computer-implemented method of claim 10, wherein performing the predetermined action includes suppressing touch input received by a capacitive touch sensor of a display device that does not have the current user focus.

14. The computer-implemented method of claim 10, wherein determining the current user focus further includes:

detecting a first ambient light level for a surrounding in front of the first display device via a first ambient light sensor and a second ambient light level for a surrounding in front of the second display device via a second ambient light sensor;

determining that the first ambient light level is above a threshold light level and that the second ambient light level is below the threshold light level;

based on at least determining that the first ambient light level is above the threshold light level, determining that the first display device has the current user focus; and

based on at least determining that the second ambient light level is below the threshold light level, determining that the second display device does not have the current user focus.

15. The computer-implemented method of claim 10, wherein determining the current user focus further includes:

detecting a first image of a surrounding in front of the first display device via a first camera and a second image of a surrounding in front of the second display device via a second camera;

processing the first image and the second image to detect facial features;

based on at least detecting facial features in the first image, determining that the first display device has the current user focus; and

based on at least detecting no facial features in the second image, determining that the second display device does not have the current user focus.

16. A mobile computing device comprising:

a processor;

two or more sensor devices including a first sensor and a second sensor, the first sensor consuming less power than the second sensor;

two or more display devices; and

a housing including the processor, the two or more sensor devices, and the two or more display devices, the housing including a hinge between a pair of display devices of the two or more display devices, the hinge being configured to permit the pair of display devices to rotate between angular orientations from a face-to-face angular orientation to a back-to-back angular orientation;

the processor being configured to;

detect a current angular orientation between the pair of display devices indicating that the pair of display devices are facing away from each other;

determine a current user focus indicating that a first display device of the pair of display devices is being viewed by the user, and that a second display device of the pair of display devices is not being viewed by the user, where to determine the current user focus, the processor is configured to;

receive a first signal from the first sensor and determine a candidate user focus based on the first signal at a first computed confidence level;

determine that the first computed confidence level does not exceed a predetermined threshold, and in response to this determination, powering up the second sensor;

receive a second signal from the second sensor;

determine the candidate user focus based on the second signal and the first signal at a second computed confidence level; and

determine whether the second computed confidence level exceeds the predetermined threshold, and if so, determine that the current user focus is the candidate user focus; and

reduce a power usage of a display device that does not have the current user focus.