Display resource management

• US 9,851,787 B2
• Filed: 11/29/2012
• Issued: 12/26/2017
• Est. Priority Date: 11/29/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Latta, Stephen, Perry, Jedd Anthony, Fleck, Rod G., Clevenger, Jack, Schaffalitzky, Frederik, Steedly, Drew, McCulloch, Daniel, McIntyre, Ian, Balan, Alexandru, Sugden, Ben, Hastings, Ryan, Mount, Brian
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Figueroa-Gibson, Gloryvid
• Application Number:
  US13/689,379
• Publication Number:
  US 20140145914A1
• Time in Patent Office: 1,853 Days
• Field of Search: 345 7- 9, 345632-633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/3215 Monitoring of peripheral devices
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A system and related methods for a resource management in a head-mounted display device are provided. In one example, the head-mounted display device includes a plurality of sensors and a display system for presenting holographic objects. A resource management program is configured to operate a selected sensor in a default power mode to achieve a selected fidelity. The program receives user-related information from one or more of the sensors, and determines whether target information is detected. Where target information is detected, the program adjusts the selected sensor to operate in a reduced power mode that uses less power than the default power mode.

Claims

1. A resource management system, comprising:

a head-mounted display device configured to be worn by a user and operatively connected to a computing device, the head-mounted display device including a plurality of sensors and a display system for presenting holographic objects, and

a resource management program executed by a processor of the computing device, the resource management program configured to;

operate a selected sensor of the plurality of sensors in a default power mode to achieve a selected level of sensor fidelity;

receive user-related information from one or more of the plurality of sensors, the user-related information selected from the group consisting of audio information, user gaze information, user location information, user movement information, user image information, and user physiological information;

determine whether target information is detected in the user-related information; and

when the target information is detected, among the plurality of sensors adjust only the selected sensor to operate in a reduced power mode that uses less power than the default power mode, thereby achieving a reduced level of sensor fidelity;

wherein the plurality of sensors includes an outward facing image sensor, an eye-tracking sensor, and a position sensor, and

wherein adjusting only the selected sensor to operate in the reduced power mode includes;

in response to detecting the target information in the user-related information received from the outward facing image sensor, one or more of reducing a refresh rate and a resolution of the outward facing image sensor,

in response to detecting the target information received from the eye-tracking sensor, reducing a refresh rate of the eye-tracking sensor, and

in response to detecting the target information received from the position sensor, reducing a sampling rate of the position sensor,

wherein the resource management system is configured to override the resource management program and adjust the selected sensor to operate in a fast response mode in response to a transition of the head-mounted display device into a new environment, the fast response mode using more power than the reduced power mode, thereby achieving an increased level of sensor fidelity.

2. The resource management system of claim 1, wherein the target information is selected from the group consisting of context-identifying audio information, a user gaze that is fixed on one of the holographic objects or the physical object, a user location at a predetermined location, movement of the user below a movement threshold, image information indicating a user body part, and awareness-related user physiological information that indicates a user awareness below an awareness threshold.

3. The resource management system of claim 1, wherein the plurality of sensors further includes a microphone.

4. The resource management system of claim 1, wherein the reduced power mode further comprises a powered-off state.

5. The resource management system of claim 1, wherein the resource management program is further configured to:

receive physical environment information from one or more of the plurality of sensors; and

determine whether the target information is detected in the physical environment information.

6. The resource management system of claim 5, wherein the physical environment information comprises physical object image data, and the resource management program is further configured to:

determine whether the outward facing image sensor has captured a threshold amount of the physical object image data; and

when the threshold amount of the physical object image data has been captured, adjust the outward facing image sensor to operate in the reduced power mode.

7. The resource management system of claim 1, wherein the resource management program is further configured to:

receive physical environment information via a network from an external source that is external to the head-mounted display device; and

based on receiving the physical environment information from the external source, adjust the selected sensor to operate in the reduced power mode.

8. The resource management system of claim 1, wherein the plurality of sensors comprises a microphone, and the resource management program is further configured to:

operate the microphone at a slower polling frequency while target audio is not detected; and

adjust the microphone to operate at a faster polling frequency that is greater than the slower polling frequency when the target audio is detected.

9. The resource management system of claim 1, wherein the target information comprises a user gaze at one of the holographic objects or a physical object, and the resource management program is further configured to:

detect the user gaze at one of the holographic objects or the physical object using the eye-tracking sensor; and

based on detecting the user gaze, reduce an illumination output of the display system of the head-mounted display device.

10. The resource management system of claim 1, wherein the target information comprises movement of the user below a movement threshold, and the resource management program is further configured to:

detect that the movement of the user is below the movement threshold using the position sensor; and

based on detecting the movement of the user below the movement threshold, adjust the position sensor to operate in the reduced power mode.

11. A method for managing resources in a head-mounted display device, comprising:

providing the head-mounted display device configured to be worn by a user and operatively connected to a computing device, the head-mounted display device including a plurality of sensors and a display system for presenting holographic objects;

operating a selected sensor of the plurality of sensors in a default power mode to achieve a selected level of sensor fidelity;

receiving user-related information from one or more of the plurality of sensors, the user-related information selected from the group consisting of audio information, user gaze information, user location information, user movement information, user image information, and user physiological information;

determining whether target information is detected in the user-related information;

when the target information is detected, among the plurality of sensors adjusting only the selected sensor to operate in a reduced power mode that uses less power than the default power mode, thereby achieving a reduced level of sensor fidelity; and

adjusting the selected sensor to operate in a fast response mode in response to a transition of the head-mounted display device into a new environment, the fast response mode using more power than the reduced power mode, thereby achieving an increased level of sensor fidelity;

wherein the plurality of sensors includes an outward facing image sensor, an eye-tracking sensor, and a position sensor, and

wherein adjusting only the selected sensor to operate in the reduced power mode includes;

in response to detecting the target information in the user-related information received from the outward facing image sensor, one or more of reducing a refresh rate and a resolution of the outward facing image sensor,

in response to detecting the target information received from the eye-tracking sensor, reducing a refresh rate of the eye-tracking sensor, and

in response to detecting the target information received from the position sensor, reducing a sampling rate of the position sensor.

12. The method for managing resources of claim 11, wherein the target information is selected from the group consisting of context-identifying audio information, a user gaze that is fixed on one of the holographic objects or a physical object, a user location at a predetermined location, movement of the user below a movement threshold, image information indicating a user body part, and awareness-related user physiological information that indicates a user awareness below an awareness threshold.

13. The method for managing resources of claim 11, further comprising:

receiving physical environment information from one or more of the plurality of sensors; and

determining whether the target information is detected in the physical environment information.

14. The method for managing resources of claim 13, wherein the physical environment information comprises physical object image data, and further comprising:

determining whether the outward facing image sensor has captured a threshold amount of the physical object image data; and

when the threshold amount of the physical object image data has been captured, adjusting the outward facing image sensor to operate in the reduced power.

15. The method for managing resources of claim 11, further comprising:

receiving physical environment information via a network from an external source that is external to the head-mounted display device; and

based on receiving the physical environment information from the external source, adjusting the selected sensor to operate in the reduced power mode.

16. The method for managing resources of claim 11, wherein the plurality of sensors comprises a microphone, and further comprising:

operating the microphone at a slower polling frequency when target audio is not detected; and

adjusting the microphone to operate at a faster polling frequency that is greater than the slower polling frequency when the target audio is detected.

17. The method for managing resources of claim 11, wherein the target information comprises a user gaze at one of the holographic objects or a physical object, and further comprising:

detecting the user gaze at one of the holographic objects or the physical object using the eye-tracking sensor; and

based on detecting the user gaze, reducing an illumination output of the display system of the head-mounted display device.

18. The method for managing resources of claim 11, wherein the target information comprises movement of the user below a movement threshold, and further comprising:

detecting that the movement of the user is below the movement threshold using the position sensor; and

based on detecting the movement of the user below the movement threshold, adjusting the position sensor to operate in the reduced power mode.

19. A resource management system, comprising:

a head-mounted display device configured to be worn by a user and operatively connected to a computing device, the head-mounted display device comprising a display system for presenting holographic objects and a plurality of sensors including an outward facing image sensor, a position sensor, an eye-tracking sensor, and a microphone; and

a resource management program executed by a processor of the computing device, the resource management program configured to;

operate a selected sensor of the plurality of sensors in a default power mode to achieve a selected level of sensor fidelity;

receive user-related information from one or more of the plurality of sensors, the user-related information selected from the group consisting of audio information, user gaze information, user location information, user movement information, user image information, and user physiological information;

receive physical environment information from one or more of the plurality of sensors;

determine whether target information is detected in the user-related information or the physical environment information; and

when the target information is detected, among the plurality of sensors adjust only the selected sensor to operate in a reduced power mode that uses less power than the default power mode, thereby achieving a reduced level of sensor fidelity;

wherein adjusting only the selected sensor to operate in the reduced power mode includes;

in response to detecting the target information in the user-related information received from the outward facing image sensor, one or more of reducing a refresh rate and a resolution of the outward facing image sensor,

in response to detecting the target information received from the eye-tracking sensor, reducing a refresh rate of the eye-tracking sensor, and

in response to detecting the target information received from the position sensor, reducing a sampling rate of the position sensor,

wherein the resource management system is configured to override the resource management program and adjust the selected sensor to operate in a fast response mode in response to a transition of the head-mounted display device into a new environment, the fast response mode using more power than the reduced power mode, thereby achieving an increased level of sensor fidelity.