IMAGE RENDERING RESPONSIVE TO USER ACTIONS IN HEAD MOUNTED DISPLAY

• US 20140361977A1
• Filed: 03/12/2014
• Published: 12/11/2014
• Est. Priority Date: 06/07/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Mao, Xiadong, Black, Glenn
• Granted Patent: US 9,897,805 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

Methods, systems, and computer programs are presented for rendering images on a head mounted display (HMD). One method includes operations for tracking, with one or more first cameras inside the HMD, the gaze of a user and for tracking motion of the HMD. The motion of the HMD is tracked by analyzing images of the HMD taken with a second camera that is not in the HMD. Further, the method includes an operation for predicting the motion of the gaze of the user based on the gaze and the motion of the HMD. Rendering policies for a plurality of regions, defined on a view rendered by the HMD, are determined based on the predicted motion of the gaze. The images are rendered on the view based on the rendering policies.

Claims

1. A method for rendering images on a head mounted display (HMD), the method comprising:

tracking, with one or more first cameras inside the HMD, a gaze of a user;

tracking motion of the HMD, wherein tracking the motion includes analyzing images of the HMD taken with a second camera that is not in the HMD;

predicting a motion of the gaze of the user based on the gaze and the motion of the HMD;

determining rendering policies for a plurality of regions defined on a view rendered by the HMD based on the predicted motion of the gaze; and

rendering images by the HMD based on the rendering policies.

2. The method as recited in claim 1, wherein predicting the motion of the gaze further includes:

tracking a trajectory of the gaze of the user over a predetermined period of time; and

tracking a trajectory of the HMD over the predetermined period of time.

3. The method as recited in claim 1, wherein the rendering policy for each region includes one or more of frequency of screen updates, rendering resolution, complexity of rendering imagery in the region, or a rendering order value that determines an order for rendering the plurality of regions.

4. The method as recited in claim 3 further including:

creating an ordered list for rendering the plurality of regions based on the rendering order values.

5. The method as recited in claim 3, wherein a region in a center of the view has a higher rendering order value than a region in a periphery of the view.

6. The method as recited in claim 1, wherein determining the rendering policies further includes:

assigning higher rendering order value to regions in a user view area defined by the gaze than to regions outside the user view area.

7. The method as recited in claim 1, wherein tracking the motion of the HMD further includes:

one or more of tracking infrared lights or tracking LED lights in the HMD.

8. The method as recited in claim 1, wherein the second camera is a stereo camera.

9. The method as recited in claim 1, wherein determining rendering policies further includes:

lowering the rendering resolution when the HMD is moving; and

increasing the rendering resolution when the HMD is stationary.

10. The method as recited in claim 1 further including:

tracking motion of a controller with the second camera; and

creating a game object that moves in synchronism with the controller, the game object being rendered on the view.

11. The method as recited in claim 10, wherein the game object is an avatar in a game that moves in synchronism with the controller.

12. The method as recited in claim 10, wherein the game object is one of a weapon, a steering device, a flashlight, a hammer, or a shield.

13. A computer program embedded in a non-transitory computer-readable storage medium, when executed by one or more processors, for rendering images on a head mounted display (HMD), the computer program comprising:

program instructions for tracking, with one or more first cameras inside the HMD, a gaze of a user;

program instructions for tracking motion of the HMD, wherein tracking the motion includes analyzing images of the HMD taken with a second camera that is not in the HMD;

program instructions for predicting a motion of the gaze of the user based on the gaze and the motion of the HMD;

program instructions for determining rendering policies for a plurality of regions defined on a view rendered by the HMD based on the predicted motion of the gaze; and

program instructions for rendering images by the HMD based on the rendering policies.

14. The computer program as recited in claim 13, wherein tracking the motion of the HMD further includes:

receiving inertial information regarding a motion of the HMD, the inertial information being captured by inertial sensors in the HMD.

15. The computer program as recited in claim 13, wherein the plurality of regions did not have a same size for all the regions.

16. A method for rendering images on a head mounted display (HMD), the method comprising:

tracking a gaze of a user wearing the HMD;

assigning rending priority values for a plurality of game objects being rendered by the HMD based on the gaze of the user, the rendering priority value for each game object defining at least one of frequency of rendering, complexity of rendering game object, image quality of game object, or rendering resolution;

changing the rendering priority values based on an importance value of each game object in the game; and

rendering by the HMD the plurality of game objects according to the rendering priority values.

17. The method as recited in claim 16, wherein the rendering priority value further includes a sorting value for ordering the rendering of the plurality of game objects.

18. The method as recited in claim 16, further including:

tracking motion of the HMD, wherein tracking the motion includes analyzing images of the HMD taken with a second camera that is not in the HMD.

19. The method as recited in claim 18 further including:

tracking motion of a controller with the second camera; and

creating a game object that moves in synchronism with the controller, the game object being rendered on the view.

20. The method as recited in claim 16, wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.