Methods for rendering interactive content to a head mounted display

• US 10,173,129 B2
• Filed: 06/04/2014
• Issued: 01/08/2019
• Est. Priority Date: 06/09/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Mikhailov, Anton, Mao, Xiaodong
• Primary Examiner(s): Torimiro, Adetokunbo O
• Application Number:
  US14/296,384
• Publication Number:
  US 20140364215A1
• Time in Patent Office: 1,679 Days
• Field of Search: 463 20, 463 22, 463 25, 463 31, 463 39, 463 42
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/803 Driving vehicles or craft, e.g. cars, airplanes, ships, robots or tanks
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G02B 2027/0112 comprising device for genereting colour display
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0176 characterised by mechanical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2320/0233 : Improving the luminance or brightness uniformity across the screen
• CPC Class Codes: G09G 2320/0626 : for control of overall brightness
• CPC Class Codes: G09G 3/001 : using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background slide projectors per se G03B23/00 = 42 HP
• CPC Class Codes: G09G 3/3406 : Control of illumination source illumination devices structurally associated with liquid crystal cells G02F1/1336
• CPC Class Codes: G09G 3/3413 : Details of control of colour illumination sources
• CPC Class Codes: H04N 13/254 : in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps

Abstract

A method for executing a game presented on a screen of a head mounted display (HMD) is provided. Execution renders interactive scenes of the game on the screen. The method includes receiving data of the HMD worn by a user to identify an HMD spatial position. The method also includes receiving data of a controller held by the user to identify a controller spatial position. During game play, the method includes rendering a point of view into the interactive scenes of the game on the screen, where the point of view is a volume of space rendered in three-dimensions. The method includes adjusting the point of view as the HMD spatial position is detected to change. The controller can drive interactivity in the game and changes in position of the controller and the HMD are correlated to provide an immersive three-dimensional game play experience.

Claims

1. A method for executing a game to be presented on a screen used for a head mounted display (HMD), comprising,

executing a game using a computer associated with the HMD, the execution rendering interactive scenes of the game on the screen;

receiving data of the HMD worn by a user to identify an HMD spatial position, the data includes image data regarding one or more light emitting diodes (LEDs) of the HMD;

receiving data of a controller held by the user to identify a controller spatial position, the data includes image data regarding an LED of the controller;

processing data by the computer for rendering on the HMD a point of view into the interactive scenes of the game on the screen used for the HMD; and

continually updating processed data by the computer for the HMD, the processed data is configured to adjust the point of view presented by the screen as the HMD spatial position is detected to change,

wherein the controller is represented in the interactive scenes to interface with a virtual object, the virtual object rendered in the interactive scenes in a virtual spatial position that relatively maps to the controller spatial position, and changes in the controller spatial position change the virtual spatial position of the interfacing with of the virtual object as viewable through the screen of the HMD, the method being executed by a processor.

2. The method of claim 1, wherein the interactive scenes are rendered in a three-dimension format, such that objects in the interactive scenes appear in front of the HMD and extend a distance into the interactive scenes.

3. The method of claim 1, wherein the HMD and controller spatial positions are tracked using one or more of image data, inertial data, wireless data, infrared data, or combinations thereof.

4. The method of claim 3, wherein tracking using image data includes detecting one or both of the HMD and controller spatial positions using a stereo camera that provides depth data.

5. The method of claim 1, wherein the computer is one of a gaming console, a personal computer, a laptop, a tablet computer, a mobile device, a cellular phone, a tablet, a thin client, a set-top box, or a media streaming device.

6. The method of claim 5, wherein the one or more LEDs of the HMD and the LED of the controller is configured to be illuminated with a color.

7. The method of claim 1, wherein the one or more LEDs of the HMD and the LED of the controller are tracked using the image data, the data of the HMD and the controller is processed by the computer associated with the HMD.

8. The method of claim 7, wherein the color is set in association with a player or mode or shade based on room lighting or background conditions.

9. A method for processing operations by a head mounted display (HMD), comprising,

the HMD is associated with a computer and a screen, the screen is located in a head attachment portion of the HMD for viewing by eyes of a user when the head attachment portion is connected to a head of the user;

processing, by the computer, interactive scenes of a computer program, the interactive scenes being rendered on the screen of the HMD;

processing, by the computer, an HMD spatial position in an area of interactivity, the HMD spatial position being determined using inertial sensor data of the HMD and by tracking at least two lights disposed on the HMD, the at least two lights configured to be tracked by a camera;

processing, by the computer, a controller spatial position in an area of interactivity, the controller spatial position being determined using at least an illuminated portion of the controller, the illuminated portion of the controller is configured to be tracked by the camera; and

during the processing of the interactive scenes, the processing of the HMD spatial position and the controller spatial position are tracked, such that changes of the HMD spatial position cause rendering of different views into the interactive scenes and changes in the controller spatial position enable movements of an interactive interface within the interactive scenes to interface with at least one virtual object rendered in the interactive scenes.

10. The method of claim 9, wherein the interactive scenes are rendered in a three-dimension format, such that objects in the interactive scenes appear in front or around of the HMD and extend different distances into the interactive scenes.

11. The method of claim 9, wherein the controller spatial position is additionally tracked using inertial data, wireless data, infrared data, or combinations thereof.

12. The method of claim 9, wherein the tracking is by way of processing image data detecting from one or both of the HMD and controller using said camera, and said camera is a stereo camera that provides at least some information regarding depth.

13. The method of claim 9, wherein the computer is communicative connected to the head mounted display.

14. The method of claim 9, wherein the lights are illuminated with a color.

15. The method of claim 14, wherein the color is set in association with a player or mode or shade based on room lighting or background conditions.

16. The method of claim 9, wherein the illuminated portion includes a light emitting diode (LED) source, and the LED source is configured to be illuminated in one or more colors.

17. The method of claim 9, wherein the computer is one of a gaming console, a personal computer, a laptop, a tablet computer, a mobile device, a cellular phone, a tablet, a thin client, a set-top box, or a media streaming device.