Tracking system for head mounted display

• US 10,684,485 B2
• Filed: 05/29/2015
• Issued: 06/16/2020
• Est. Priority Date: 03/06/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeff, Mallinson, Dominic, Osman, Steven
• Primary Examiner(s): Tswei, YuJang
• Application Number:
  US14/726,341
• Publication Number:
  US 20160260251A1
• Time in Patent Office: 1,845 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method of tracking a location of a head mounted display and generating additional virtual reality scene data to provide the user with a seamless virtual reality experience as the user interacts with and moves relative to the virtual reality scene. An initial position and pose of the HMD is determined using a camera or similar sensor mounted on or in the HMD. As the HMD is moved into a second position and pose, images of two or more fixed points are captured by the camera or sensor to determine a difference in position and pose of the HMD. The difference in position and pose of the HMD is used to predict corresponding movement in the virtual reality scene and generate corresponding additional virtual reality scene data for rendering on the HMD.

Claims

1. A method for tracking a head mounted display (HMD) used for rendering a virtual reality scene, the HMD including a display screen for rendering the virtual reality scene, the method comprising:

capturing image data using an infrared camera facing out of an external surface of the HMD, the image data capturing a real space in which the HMD is located;

processing frames of the image data to identify at least two points of light projected from an infrared emitter that is fixed in the real space and is separate from said HMD upon a surface in the real space as reference points, the real space having real-world objects disposed in and around an area of interactivity and the at least two points of light;

identifying changes in location of the at least two points of light in the captured image data, the changes in location in compared frames of said image data being associated to position and pose changes by the HMD in the real space;

wherein the position and pose changes are configured to automatically control rendering adjustments to the virtual reality scene presented on the display screen of the HMD, the adjustments including changes in a view perspective into of the virtual reality scene and rendering of additional content for the virtual reality scene;

wherein said additional content of the virtual reality scene is rendered for said changes in the view perspective into the virtual reality scene based on analysis of the points of light captured by the infrared camera facing out of the external surface of the HMD.

2. The method of claim 1, wherein the capturing is performed at a frame rate that continues while the tracking of the HMD is occurring.

3. The method of claim 1, wherein the processing is performed by a computing device that is connected to the HMD.

4. The method of claim 1, wherein the position and pose changes are determined by a computing device that is connected to the HMD and the computing device is configured to control the rendering of adjustments to the virtual reality scene rendered on the display screen of the HMD.

5. The method of claim 1, further comprising, additionally one or more of a red-green-blue (RGB) camera, a video camera, a depth camera, or a position sensing device (PSD) integrated on the external surface of the HMD.

6. The method of claim 1, wherein the infrared camera and the external surface is part of housing of the HMD or a strap of the HMD.

7. The method of claim 1, further comprising,

examining inertial data of the HMD during the position and pose changes in the real space, the inertial data generated on the HMD to provide an additional tracking variable usable when automatically controlling rendering adjustments.

8. A head mounted display (HMD), comprising,

a housing including a screen for displaying images associated with a virtual reality scene;

a infrared camera facing out of an external surface of the housing; and

a processor for controlling capturing of image data by the infrared camera, the image data capturing a real space in which the HMD is located along with at least two points of light detected to be projected by an infrared emitter onto a surface of the real space as reference points, wherein the infrared emitter is located in the real space at a fixed location that is separate from the HMD,

the real space having real-world objects disposed in and around an area of interactivity and the at least two points of light, the processor is configured to continually transfer the image data to a computing device during position tracking of the HMD in the real space, the processor configured to identify changes in position and pose of the HMD based on changes in location of the at least two points of light in compared frames of the image data, the processor is further configured to receive content for the virtual reality scene to be rendered on the screen based on the identified changes in position and pose;

wherein the changes in position and pose cause automatic adjustments to a view perspective into of the virtual reality scene and rendering of additional content for the virtual reality scene;

wherein said additional content of the virtual reality scene is rendered for changes in the view perspective into the virtual reality scene based on analysis of said at least two points of light captured by the infrared camera.

9. The HMD of claim 8, further comprising,

an inertial sensor integrated with the housing, the inertial sensor providing motion data to the processor for determining one or more of position, position and pose or acceleration.

10. The HMD of claim 8, further comprising, additionally one or more of a camera, or a red-green-blue (RGB) camera, or a video camera, or a depth camera, or a position sensing device (PSD) disposed proximate to an external surface of the housing.

11. The method of claim 8, wherein the emitter is directed toward the surface.