Calibration of portable devices in a shared virtual space

• US 9,513,700 B2
• Filed: 04/23/2014
• Issued: 12/06/2016
• Est. Priority Date: 12/24/2009
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Weising, George, Miller, Thomas
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): LAM, VINH TANG
• Application Number:
  US14/260,208
• Publication Number:
  US 20140232652A1
• Time in Patent Office: 958 Days
• Field of Search: 345/158
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2457 Display screens, e.g. monitors, video displays
• CPC Class Codes: A63F 3/02 Chess; Similar board games
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0261 : in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 5/08 : Cursor circuits
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors

Abstract

Methods, systems, and computer programs are provided for generating an interactive space. One method includes operations for associating a first device to a reference point in 3D space, and for calculating by the first device a position of the first device in the 3D space based on inertial information captured by the first device and utilizing dead reckoning. Further, the method includes operations for capturing images with a camera of the first device, and for identifying locations of one or more static features in the images. The position of the first device is corrected based on the identified locations of the one or more static features, and a view of an interactive scene is presented in a display of the first device, where the interactive scene is tied to the reference point and includes virtual objects.

Claims

1. A method comprising:

associating a first device to a reference point in a three-dimensional (3D) space;

calculating by the first device a position of the first device in the 3D space based on inertial information captured by the first device, wherein calculating the position uses dead reckoning by estimating a current position of the first device based upon a previously determined position and advancing the previously determined position based upon a speed of the first device over elapsed time and course;

capturing images with a camera of the first device;

identifying locations of one or more static features in the images;

correcting the position of the first device based on the identified locations of the one or more static features, the correcting further includes attributing a difference between a first estimate calculated based on inertial information and a second estimate, calculated based on the identified locations of the one or more static features, to a dead reckoning error; and

presenting a view of an interactive scene in a display of the first device, wherein operations of the method are executed by a processor.

2. The method as recited in claim 1, wherein calculating the position is performed at predetermined intervals or after a predetermined number of inertial measurements.

3. The method as recited in claim 1, wherein correcting the position further includes:

adjusting the first estimate to have an expected location of a first static feature be equal to a measured location of the first static feature.

4. The method as recited in claim 1, wherein the interactive scene is tied to the reference point and includes virtual objects.

5. The method as recited in claim 1, wherein the view shows all or part of the interactive scene as observed from a current location of the first device.

6. The method as recited in claim 1, wherein the one or more static features include one or more of an edge of table, or a window, or a television, or a light source, or a painting on a wall.

7. A method comprising:

associating a first device to a reference point in a three-dimensional (3D) space;

exchanging position information between the first device and a second device to enable the second device to associate to the reference point;

calculating by the first device a position of the first device in the 3D space based on inertial information captured by the first device, the calculating by the first device the position of the first device further includes utilizing dead reckoning by estimating a current position of the first device based upon a previously determined position and advancing the previously determined position to the current position;

capturing images with a camera of the first device;

identifying locations of one or more static features in the images;

correcting the position of the first device based on the identified locations of the one or more static features and inertial information, the correcting is applied to a dead reckoning error; and

presenting a view of a shared interactive scene in a display of the first device, wherein operations of the method are executed by a processor.

8. The method as recited in claim 7, wherein the first device and a second device exchange dead reckoning data and static feature information.

9. The method as recited in claim 7, further including:

obtaining by the first device a relative position of the second device utilizing WiFi triangulation, player face detection, or identification of the second device within the captured images.

10. The method as recited in claim 7, further including:

obtaining by the first device a relative position of the second device utilizing audio triangulation by utilizing ultrasonic communication and one or more directional microphones.

11. The method as recited in claim 7, wherein the view in the first device is independent from the view in the second device, the view in the first device being based on a relative position of the first device with respect to the shared interactive scene.

12. The method as recited in claim 7, further including:

determining a relative position of the second device with reference to the first device based on image analysis to identify a geometry of the second device in the images.

13. The method as recited in claim 7, wherein the view shows all or part of the shared interactive scene as observed from a current location of the first device.

14. The method as recited in claim 7, wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

15. A non-transitory computer-readable storage medium storing a computer program, the computer-readable storage medium comprising:

program instructions for associating a first device to a reference point in a three-dimensional (3D) space;

program instructions for calculating by the first device a position of the first device in the 3D space based on inertial information captured by the first device, wherein the calculating by the first device the position of the first device further includes utilizing dead reckoning by estimating a current position of the first device based upon a previously determined position and advancing the previously determined position based upon a speed of the first device over elapsed time and course;

program instructions for capturing images with a camera of the first device;

program instructions for identifying locations of one or more static features in the images;

program instructions for correcting the position of the first device based on the identified locations of the one or more static features, the correcting further includes attributing a difference between a first estimate calculated based on inertial information and a second estimate, calculated based on the identified locations of the one or more static features, to a dead reckoning error; and

program instructions for presenting a views of an interactive scene in a display of the first device, wherein the interactive scene is tied to the reference point and includes virtual object.