Automatic view adjustment

• US 9,658,688 B2
• Filed: 10/15/2013
• Issued: 05/23/2017
• Est. Priority Date: 10/15/2013
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Li, Liqun, Shen, Guobin
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): MATHEWS, CRYSTAL
• Application Number:
  US14/054,111
• Publication Number:
  US 20150102995A1
• Time in Patent Office: 1,316 Days
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/144 the light being ambient light
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A view adjustment system using information captured by one or more sensors on a client device determines a projection direction for content to be displayed on a display of the client device. Upon determining the projection direction, the view adjustment system transforms the content into a perspective view based on the determined projection direction and prompts the client device to present the content in the perspective view to a user. The view adjustment system may monitor changes in relative position and/or direction of the user with respect to the display, adjust the projection direction, and transform the content to reflect these changes.

Claims

1. A system comprising:

at least one processor;

memory operatively coupled to the at least one processor;

a first sensor operatively coupled to the at least one processor;

a second sensor operatively coupled to the at least one processor;

a display operatively coupled to the at least one processor;

a detection module stored in the memory and executable by the at least one processor to estimate an angle of view between an eye of a user and a surface of the display

a transformation module stored in the memory and executable by the at least one processor to transform content for display by the display into a perspective view, the perspective view projecting the content into a plane perpendicular to a line of sight of the user based at least on the estimated angle of view;

a display module stored in the memory and executable by the at least one processor to cause presentation of the content in the perspective view on the display; and

a view adjustment module stored in the memory and executable by the at least one processor to;

determine whether the system is being used in a hand-held state or an off-the-body state;

determine that a condition for estimating the angle of view is satisfied, the condition being associated with one of the hand-held state or the off-the-body state, wherein determining that the condition for estimating the angle of view is satisfied includes;

in response to determining that the system is in the hand-held state, determining that input received from the first sensor indicates a change in orientation of the device greater than or equal to a threshold change in orientation, the threshold change in orientation being greater than zero change in orientation; and

in response to determining that the system is in the off-the-body state, determining that input received from the second sensor of the device indicates a change in light incident the display of the device greater than or equal to a threshold change in light incident the display, the threshold change in light incident the display being greater than zero change in light incident the display;

based at least on determining that the condition is satisfied, trigger the transformation module to re-transform the content for display based at least on the threshold change in orientation or the threshold change in lights.

2. The system of claim 1, wherein the image sensor comprises at least one of a fisheye camera, a wide field-of-view camera, or an array of cameras.

3. A device comprising:

a processor;

an accelerometer;

a light sensor;

a display; and

a memory having stored thereon instructions executable by the at least one processor that, when executed by the at least one processor, configure the system to perform operations including;

determining whether the device is being used in a hand-held state or an off-the-body state;

determining that a condition for estimating an angle of view is satisfied, the condition being associated with one of the hand-held state or the off-the-body state, wherein determining that the condition for estimating the angle of view is satisfied includes;

in response to determining that the device is in the hand-held state, determining that input received from the accelerometer of the device indicates a change in orientation of the device greater than or equal to a threshold change in orientation, the threshold change in orientation being greater than zero change in orientation; and

in response to determining that the device is in the off-the-body state, determining that input received from the light sensor of the device indicates a change in light incident the display of the device greater than or equal to a threshold change in light incident the display, the threshold change in light incident the display being greater than zero change in light incident the display;

in response to determining that the condition is satisfied, estimating the angle of view between a user and the display of the device; and

transforming content displayed on the display of the device into a perspective view based at least on the estimated angle of view.

4. The device of claim 3, further comprising presenting the content in the perspective view to the user.

5. The device of claim 4, further comprising monitoring the angle of view between the user and the display of the device.

6. The device of claim 3, wherein the transforming comprises projecting the content on a two-dimensional plane that is perpendicular to a line of sight of the user that is determined based on the estimated angle of view.

7. The device of claim 3, wherein the device comprises a computing device, a mobile device, or a display device.

8. The device of claim 3, further comprising:

detecting a change in the angle of view between the user and the display of the device; and

adjusting the perspective view based on the change in the angle of view.

9. The device of claim 3, wherein the condition comprises at least one of:

the display being turned on;

the user using the device;

or

the user being within a predefined proximity of the device.

10. The device of claim 3, wherein the estimating comprises:

detecting, by a sensor, a face of the user;

identifying an eye of the user from the detected face;

determining a line of sight from the eye of the user to the display of the device; and

determining the angle of view based on the estimated line of sight.

11. The device of claim 10, wherein the sensor comprises at least one of a fisheye camera, a wide field-of-view camera, or an array of cameras.

12. The device of claim 3, wherein the angle of view corresponds to an angle between a line of sight from the user and a plane parallel to a surface of the display of the device.

13. One or more computer storage media storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

receiving input from a first sensor;

detecting a change in a viewing condition based at least in part on the input received from the first sensor;

determining whether the change in the viewing condition is greater than, equal to, or less than a predetermined threshold;

in response to determining that the change in the viewing condition is less than the predetermined threshold, maintaining a projection direction for displaying content on a display;

in response to determining that the change in the viewing condition is greater than or equal to the predetermined threshold;

obtaining input from a second sensor different than the first sensor, wherein the second sensor comprises a fisheye camera or a wide field-of-view camera;

determining a line of sight from a user to the display based on the input obtained from the second sensor; and

transforming content displayed on the display into a perspective view that projects the content into a plane substantially perpendicular to the line of sight from the user.

14. The one or more computer storage media of claim 13, the acts further comprising providing the content in the perspective view for presentation to the user.

15. The one or more computer storage media of claim 13, the acts further comprising determining that a condition for determining the line of sight is satisfied.

16. The one or more computer storage media of claim 15, wherein the condition comprises at least one of:

the display being turned on;

the user using a device of the display;

the user being within a predefined proximity of the display;

or a change in an amount of light incident on the device of the display.

17. The one or more computer storage media of claim 13, the acts further comprising adjusting the input based on one or more distortions associated with the fisheye camera or the wide field-of-view camera.

18. The one or more computer storage media of claim 13, wherein the first sensor consumes less power than the second sensor.

19. The one or more computer storage media of claim 13, wherein the first sensor comprises at least one of a light sensor, a proximity sensor, an accelerometer, or a compass.