Tracking position of device inside-out for virtual reality interactivity

• US 10,535,153 B2
• Filed: 12/06/2016
• Issued: 01/14/2020
• Est. Priority Date: 12/24/2009
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Weising, George, Miller, Thomas
• Primary Examiner(s): Lam, Vinh T
• Application Number:
  US15/371,151
• Publication Number:
  US 20170084051A1
• Time in Patent Office: 1,134 Days
• Field of Search: 345158, 345633, 345474, 345 11, 345 23
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2457 Display screens, e.g. monitors, video displays
• CPC Class Codes: A63F 3/02 Chess; Similar board games
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0261 : in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 5/08 : Cursor circuits
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors

Abstract

Methods, systems, and computer programs are provided for generating an interactive space. One method includes operations for associating a first device to a reference point in 3D space, and for calculating by the first device a position of the first device in the 3D space based on inertial information captured by the first device and utilizing dead reckoning. Further, the method includes operations for capturing images with a camera of the first device, and for identifying locations of one or more static features in the images. The position of the first device is corrected based on the identified locations of the one or more static features, and a view of an interactive scene is presented in a display of the first device, where the interactive scene is tied to the reference point and includes virtual objects.

Claims

1. A method comprising:

synchronizing a device to a reference point in a three-dimensional (3D) space, wherein the 3D space is associated with a first coordinate system and the reference point is viewable via a camera of the device at synchronization and one or more other times after synchronization to relocate and use the reference point, where synchronization includes calibrating to an origin point of the device to the reference point;

determining by the device a first position and a first orientation of the device within the 3D space based on inertial information captured by the device;

generating by the device a virtual reality (VR) space based on the reference point, wherein the VR space includes virtual objects, wherein a second coordinate system of the VR space is based on the first coordinate system of the 3D space such that the first position within the 3D space is translated to a second position in the VR space; and

presenting a first view of the VR space based on the first position of the device and the first orientation of the device relative to the reference point, such that the first view and changes in the first view are independent of a character in the VR space.

2. The method of claim 1, wherein the first position and first orientation are based on at least one image captured from the camera of the device.

3. The method of claim 1, further comprising:

capturing one or more images of the 3D space with the camera of the device;

identifying locations of one or more static features in the one or more images; and

correcting the first position and first orientation of the device based on identified locations of the one or more static features.

4. The method of claim 1, wherein the 3D space comprises a real-world environment.

5. The method of claim 1, further comprising:

determining a second position and a second orientation of the device in the 3D space based on the inertial information; and

presenting a second view of the VR space based on the second position and the second orientation of the device.

6. The method of claim 1, wherein the device is taken from a group consisting essentially of:

a portable gaming device;

a handheld portable gaming device;

a game controller;

a cell phone;

a head mounted display;

a notebook;

a tablet;

a netbook; and

a PDA.

7. The method of claim 1, wherein the synchronizing comprises at least one of the following:

tapping the device;

placing the first device at the reference point;

providing an input to a user interface; and

engaging an actuator.

8. A method comprising:

synchronizing a device to a reference point in a three-dimensional (3D) space, wherein the 3D space is associated with a first coordinate system, and the reference point is viewable via a camera of the device at synchronization and one or more other times after synchronization to relocate and use the reference point, where synchronization includes calibrating to an origin point of the device to the reference point;

determining by the device a first position and a first orientation of the device within the 3D space based on inertial information captured by the device;

generating by the device an augmented reality (AR) space based on the reference point, wherein the AR space includes virtual objects and real objects, wherein a second coordinate system of the AR space is based on the first coordinate system of the 3D space such that a first position within the 3D space is translated to a second position in the AR space; and

presenting a first view of the AR space based on the first position of the device and the first orientation of the device relative to the reference point, such that the first view and changes in the first view are independent of a character in the VR space.

9. The method of claim 8, further comprising:

capturing one or more images of the 3D space with the camera of the device;

identifying locations of one or more static features in the one or more images; and

correcting the first position and first orientation of the device based on identified locations of the one or more static features.

10. The method of claim 8, wherein the 3D space comprises a real-world environment.

11. The method of claim 8, wherein the device is taken from a group consisting essentially of:

a portable gaming device;

a handheld portable gaming device;

a game controller;

a cell phone;

a head mounted display;

a notebook;

a tablet;

a netbook; and

a PDA.

12. A portable handheld device for creating virtual reality, the portable handheld device comprising:

a camera and inertial system for synchronizing the device to a reference point in a three-dimensional (3D) space, and the reference point is viewable via the camera of the device at synchronization and one or more other times after synchronization to relocate and use the reference point, where synchronization includes calibrating to an origin point of the device to the reference point, wherein the 3D space is associated with a first coordinate system, wherein the camera and inertial system is further configured for determining a first position of the device and a first orientation of the device in the 3D space based on inertial information captured by the inertial system of the device;

a processor of the device for generating a virtual reality (VR) space based on the reference point, wherein the VR space includes virtual objects, wherein a second coordinate system of the VR space is based on the first coordinate system of the 3D space such that a first position within the 3D space is translated to a second position in the VR space, wherein the processor is further configured for generating a first view of the VR space based on the first position of the device and the first orientation of the device relative to the reference point, such that the first view and changes in the first view are independent of a character in the VR space; and

a display for presenting the first view of the VR space.

13. The portable handheld device of claim 12,

wherein the camera and inertial system is further configured for tracking the device within the 3D space, and for determining a second position and a second orientation of the device in the 3D space based on the inertial information, and

wherein the processor is further configured for presenting a second view of the VR space based on the second position of the device and the second orientation of the device.

14. The portable handheld device of claim 12, wherein the camera and inertial system includes at least one location tracking module for capturing the inertial information, wherein the at least one location tracking module is taken from a group consisting of:

a 3-axis accelerometer;

a 3 axis gyroscope;

a camera;

a GPS module; and

a depth camera.

15. The portable handheld device of claim 12, further comprising:

the camera being used for capturing one or more images of the 3D space,

wherein the camera and inertial system is further configured for identifying locations of one or more static features in the one or more images, and correcting the first position and first orientation of the device based on identified locations of the one or more static features.

16. The portable handheld device of claim 12, wherein the portable handheld device is taken from a group consisting essentially of:

a portable gaming device;

a handheld portable gaming device;

a game controller;

a cell phone;

a head mounted display;

a notebook;

a tablet;

a netbook; and

a PDA.