Tracking System for Head Mounted Display

• US 20160260251A1
• Filed: 05/29/2015
• Published: 09/08/2016
• Est. Priority Date: 03/06/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeff, Mallinson, Dominic, Osman, Steven
• Granted Patent: US 10,684,485 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method of tracking a location of a head mounted display and generating additional virtual reality scene data to provide the user with a seamless virtual reality experience as the user interacts with and moves relative to the virtual reality scene. An initial position and pose of the HMD is determined using a camera or similar sensor mounted on or in the HMD. As the HMD is moved into a second position and pose, images of two or more fixed points are captured by the camera or sensor to determine a difference in position and pose of the HMD. The difference in position and pose of the HMD is used to predict corresponding movement in the virtual reality scene and generate corresponding additional virtual reality scene data for rendering on the HMD.

Claims

1. A system for processing location content for a head mounted display (HMD), comprising:

a computing device interfaced with a user input device and the HMD, the computing device including;

an input device interface for receiving input from a user input device or the HMD, the input device interface processes instructions to select content for display in the HMD;

a virtual reality space generation module for rendering the selected content in the HMD in a form of a virtual reality scene; and

an HMD movement module for tracking movement of the HMD in a real space and for identifying a first position and pose of the HMD in the real space, the HMD movement module providing HMD position and pose change data to the virtual reality space generation module for rendering additional content of the virtual reality scene corresponding to a second position and pose of the HMD in the real space; and

a real space mapping system coupled to the computing device, the real space mapping system including;

a light source for projecting a plurality of points of light across at least a portion of the real space; and

a camera integrated with the HMD for capturing an image of the plurality of points of light;

wherein the second position and pose is determined by the HMD movement module using the plurality of points from the image captured by the camera.

2. The system of claim 1, wherein the HMD movement module is capable of analyzing the image captured by the camera to identify a plurality of fixed points in the real space.

3. The system of claim 2, wherein providing HMD position and posechange data includes:

identifying the HMD to the first position and pose;

selecting at least two of the plurality of fixed points;

determining a first relative distance between the selected fixed points with the HMD in the first position and pose;

identifying the HMD to the second position and pose;

determining a second relative distance between the selected fixed points with the HMD in the second position and pose; and

comparing the first relative distance and the second relative distance to determine the HMD position and pose change data equal to a difference between the first position and pose and the second position and pose.

4. The system of claim 2, wherein the virtual reality space generation module is configured to continually generate the additional content for the HMD depending on at least one position and pose that the HMD is moved toward while rendering the virtual reality scene.

5. The system of claim 2, wherein analysis of the image or a plurality of images captured by the camera is used to detect a location of the HMD in real space, wherein the location of the HMD in real space is translated to a location of the HMD in the virtual reality scene, wherein the HMD is used to identify interactions between the user input device and a virtual object in the virtual reality scene.

6. The system of claim 2, wherein the plurality of fixed points include at least a portion of the plurality of points of light.

7. The system of claim 2, wherein the plurality of fixed points include at least a portion of a plurality of identified points disposed on one or more fixed objects present in the real space.

8. The system of claim 1, wherein the content selected for display in the HMD is accessed from one or both of a local storage of the computing device or over a network.

9. The system of claim 1, wherein the light source is coupled to a fixed location separate from the HMD.

10. The system of claim 1, wherein the light source projects the plurality of points of light in a visible spectrum light.

12. The system of claim 1, wherein the light source projects the plurality of points of light across at least the portion of the real space as part of a calibration operation, wherein at least one image of the plurality of points of light are saved as a reference image.

13. The system of claim 1, further comprising,

at least one inertial sensor disposed in the HMD, the inertial sensor generating motion data usable by the computing device to determine a corresponding position and pose of the HMD for use in viewing the virtual reality scene;

wherein the corresponding position and pose is used to supplement positions and poses determined by HMD position and pose change data using the real space mapping system.

14. The method of claim 13, wherein the at least two points of light projected upon the surface in the real space are projected by a light emitter.

15. The method of claim 13, wherein the light emitter is located in the real space at a fixed location that is separate from the HMD.

16. The method of claim 13, wherein the light emitter is integrated with the HMD.

17. The method of claim 13, wherein continuing the capturing is performed at a frame rate that continues while the tracking of the HMD is occurring.

18. The method of claim 13, wherein the processing is performed by a computing device that is connected to the HMD.

19. The method of claim 13, wherein the position and pose changes are determined by a computing device that is connected to the HMD and the computing device is configured to control the rendering of adjustments to the virtual reality scene rendered on the display screen of the HMD.

20. The method of claim 13, wherein the at least one device integrated on the external surface of the HMD for capturing image data is one of a red-green-blue (RGB) camera, an infrared (IR) camera, a video camera, a depth camera, or a position sensing device (PSD).

21. The method of claim 13, wherein the at least one device is a camera and the external surface is part of housing of the HMD or a strap of the HMD.

22. The method of claim 13, further comprising,

examining inertial data of the HMD during the position and pose changes in the real space, the inertial data generated on the HMD to provide an additional tracking variable usable when automatically controlling rendering adjustments.

13-1. A method for tracking a head mounted display (HMD) used for rendering a virtual reality scene, the HMD including a display screen for rendering the virtual reality scene, the method comprising:

capturing image data using at least one device integrated on an external surface of the HMD, the image data capturing a real space in which the HMD is located;

processing the image data to identify at least two points of light projected upon a surface in the real space;

continuing the capturing and the processing to identify changes in location of the at least two points of light in the captured image data, the changes in location identifying position and pose changes by the HMD in the real space;

wherein the position and pose changes are configured to automatically control rendering adjustments to the virtual reality scene rendered on the display screen of the HMD, the adjustments including one or both of changes in a view perspective into of the virtual reality scene and rendering of additional content for the virtual reality scene.

23. A head mounted display (HMD), comprising,

a housing including a screen for displaying images associated with a virtual reality scene;

a sensing device integrated on an external surface of the housing; and

a processor for controlling capturing of image data by the sensing device, the image data capturing a real space in which the HMD is located along with at least two points of light detected to be projected by an emitter onto a surface of the real space, the processor is configured to continually transfer the image data to a computing device during position tracking of the HMD in the real space, the processor configured to identify changes in position and position and pose of the HMD based on changes in location of the at least two points of light in the image data, the processor is further configured to receive content for the virtual reality scene to be rendered on the screen based on the identified changes in position and position and pose;

wherein the changes in position and position and pose cause automatic adjustments to one or both of a view perspective into of the virtual reality scene and rendering of additional content for the virtual reality scene.

24. The HMD of claim 23, further comprising,

an inertial sensor integrated with the housing, the inertial sensor providing motion data to the processor for determining one or more of position, position and pose or acceleration.

25. The HMD of claim 23, wherein the sensing device is one of a camera, or a red-green-blue (RGB) camera, or an infrared (IR) camera, or a video camera, or a depth camera, or a position sensing device (PSD).

26. The HMD of claim 23, wherein the emitter is a light emitter, or a laser light emitter, or a infrared emitter, or a visible spectrum energy emitter.

27. The method of claim 23, wherein the emitter is located in the real space at a fixed location that is separate from the HMD, and is directed toward the surface.

28. The method of claim 23, wherein the emitter is integrated with the HMD.