Image Rendering Responsive To User Actions In Head Mounted Display

• US 20180188534A1
• Filed: 02/20/2018
• Published: 07/05/2018
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Granted Patent: US 10,545,338 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

Methods, systems, and computer programs are presented for rendering images on a head mounted display (HMD). One method includes operations for tracking, with one or more first cameras inside the HMD, the gaze of a user and for tracking motion of the HMD. The motion of the HMD is tracked by analyzing images of the HMD taken with a second camera that is not in the HMD. Further, the method includes an operation for predicting the motion of the gaze of the user based on the gaze and the motion of the HMD. Rendering policies for a plurality of regions, defined on a view rendered by the HMD, are determined based on the predicted motion of the gaze. The images are rendered on the view based on the rendering policies.

Claims

1. A method for rendering images on a display of a head mounted display (HMD), comprising:

tracking, with a gaze detection module inside the HMD, a gaze of a user of the HMD for generating gaze data;

predicting, based on the gaze data, that the gaze of the user is to be directed toward a first region of the display of the HMD at a future time; and

rendering the first region of the display with a higher image quality than than regions away from the first region before the gaze of the user arrives at the first region.

2. The method of claim 1, wherein the tracking the gaze of the user includes tracking movements of eyes of the user, and wherein the gaze data includes data related the movements of the eyes of the user.

3. The method of claim 1, wherein the gaze data is usable to calculate a trajectory of the gaze.

4. The method of claim 3, wherein at least part of the trajectory of the gaze is usable for the predicting that the gaze of the user is to be directed toward the first region of the display of the HMD at the future time.

5. The method of claim 1, further comprising:

determining that a virtual object is to be displayed in the first region;

wherein the predicting that the gaze of the user is to be directed toward the first region is further based on said determining that the virtual object is to be displayed in the first region.

6. The method of claim 1, wherein the gaze detection module includes one or more cameras, the one or more cameras are configured to capture movements of eyes of the user.

7. The method of claim 1, further comprising:

tracking motion data of the HMD;

wherein the predicting that the gaze of the user is to be directed toward the first region is further based on the tracking of the motion data of the HMD.

8. The method of claim 1, wherein the first region demarcates an area of the display for the higher image quality.

9. The method of claim 8, wherein the regions away from the first region define portions of the display that are not included within the area demarcated by the first region, the portions of the display that are not included within the area demarcated by the first region are rendered with a lower image quality.

10. The method of claim 1, wherein image quality is defined by one or more of a resolution, or a complexity of rendering imagery, or a scanning out rate, or a screen update frequency.

11. A method for rendering images on a display of a head mounted display (HMD), comprising:

tracking, with a gaze detection module, a gaze of a user of the HMD for generating gaze data;

accessing importance values for a plurality of virtual objects to be displayed by the HMD;

predicting, based on the gaze data and based on the importance values for the plurality of virtual objects, that the gaze of the user is to be directed toward a first region of the display at a future time;

assigning rendering priority values for the plurality of virtual objects based on a proximity between each of the plurality of virtual objects and the first region; and

rendering, by the HMD, the plurality of virtual objects to the display according to the rendering priority values before the gaze of the user arrives at the first region.

12. The method of claim 11, wherein virtual objects that are more proximal to the first region are assigned higher rendering priority values than virtual objects that are less proximal to the first region.

13. The method of claim 11, wherein the tracking the gaze of the user includes tracking movements of eyes of the user, and wherein the gaze data is related to the movements of the eyes of the user.

14. The method of claim 11, wherein the at least part of the gaze data is usable to calculate a trajectory of the gaze, and wherein the trajectory of the gaze is usable for said predicting that the gaze of the user is to be directed toward the first region of the display at the future time.

15. The method of claim 11, wherein the gaze detection module includes one or more cameras for capturing movements of eyes of the user for said generating the gaze data.

16. The method of claim 11, further comprising:

tracking motion data of the HMD;

wherein the predicting that the gaze of the user is to be directed toward the first region of the display at the future time is further based on the tracking of the motion data of the HMD.

17. The method of claim 11, wherein the rendering priority values define one or more of a frequency of rendering, or a complexity of rendering, or an image quality, or a resolution.

18. A computer program embedded in a non-transitory computer-readable storage medium, when executed by one or more processors for rendering images on a display of a head mounted display (HMD), the computer program comprising:

program instructions for tracking, with a gaze detection module inside the HMD, a gaze of a user for generating gaze data, the tracking the gaze includes tracking movements of eyes of the user;

program instructions for predicting, based on the gaze data, that the gaze of the user is to be directed toward a first region of the display of the HMD at a future time; and

program instructions for rendering the first region of the display with an image quality that is greater than that of regions away from the first region before the gaze of the user arrives at the first region.

19. The computer program as recited in claim 18, further comprising:

program instructions for accessing an importance value of a virtual object that is to be displayed within the first region;

wherein predicting that the gaze of the user is to be directed toward the first region of the display at a future time is further based on the importance value of the virtual object that is to be displayed within the first region.

20. The computer program as recited in claim 18, wherein the first region demarcates an area of the display for the higher image quality.

21. The computer program as recited in claim 20, wherein the regions away from the first region defined portions of the display that are not included within the area demarcated by the first region, the portions of the display not includes within the area demarcated by the first region are rendered with a lower image quality.

22. The computer program as recited in claim 20, wherein the image quality is defined by one or more of a resolution, or a complexity of imagery, or a scanning out rate, or a screen update frequency.