Calibration of portable devices in a shared virtual space

• US 8,537,113 B2
• Filed: 12/20/2010
• Issued: 09/17/2013
• Est. Priority Date: 03/05/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Weising, George, Miller, Thomas
• Primary Examiner(s): Wang, Quan-Zhen
• Assistant Examiner(s): LAM, VINH TANG
• Application Number:
  US12/973,827
• Publication Number:
  US 20110216002A1
• Time in Patent Office: 1,002 Days
• Field of Search: 345/158, 436 1- 8, 436 30- 42, 715810-856
• US Class Current: 345/158
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2457 Display screens, e.g. monitors, video displays
• CPC Class Codes: A63F 3/02 Chess; Similar board games
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0261 : in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 5/08 : Cursor circuits
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors

Abstract

Methods, systems, and computer programs for generating an interactive space viewable through at least a first and a second device are presented. The method includes an operation for detecting from the first device a location of the second device or vice versa. Further, synchronization information data is exchanged between the first and the second device to identify a reference point in a three-dimensional (3D) space relative to the physical location of the devices in the 3D space. The devices establish the physical location in the 3D space of the other device when setting the reference point. The method further includes an operation for generating views of an interactive scene in the displays of the first and second devices. The interactive scene is tied to the reference point and includes virtual objects. The view in the display shows the interactive scene as observed from the current location of the corresponding device. Moving the device in the 3D space causes the view to change according to the perspective from the current location.

Claims

1. A method for generating an interactive space viewable through at least a first and a second device, the method comprising:

detecting from the first device a location of the second device or detecting from the second device a location of the first device, wherein the first device and the second device are handheld devices;

exchanging synchronization information data between the first and second devices to identify a reference point in a three-dimensional (3D) space relative to a physical location of the first and second devices in the 3D space, wherein the first and second devices establish the physical location in the 3D space of the other device when setting the reference point; and

generating views of an interactive scene in corresponding displays of the first and second devices, wherein the corresponding device acts as a camera viewing into the interactive scene, the interactive scene is tied to the reference point and includes virtual objects, each view showing all or part of the interactive scene and showing the interactive scene as observed from a current location of the corresponding device with a viewing angle of the corresponding camera, wherein device movement in the 3D space causes the corresponding view to change according to the current location and the viewing angle.

2. The method as recited in claim 1, wherein detecting from the first device the location of the second device further includes:

detecting that the first device has tapped a first object at a first time;

receiving information that the second device has tapped a second object at a second time; and

determining that the first device has tapped the second device when the first and the second times are substantially simultaneous.

3. The method as recited in claim 1, wherein detecting from the first device the location of the second device further includes:

recognizing the second device in an image taken with a camera in the first device.

4. The method as recited in claim 1, wherein detecting from the first device the location of the second device further includes:

detecting a light source in the second device in an image taken with a camera in the first device.

5. The method as recited in claim 1, wherein detecting from the first device the location of the second device further includes:

detecting a distance between the first device and the second using a depth camera.

6. The method as recited in claim 1, wherein detecting from the first device the location of the second device further includes:

recognizing a display of the second device in an image taken with a camera in the first device.

7. The method as recited in claim 1, wherein detecting from the first device the location of the second device further includes:

recognizing a pattern in the display of the second device in an image taken with a camera in the first device.

8. The method as recited in claim 1, further including:

tracking motion of the first device by the first device, wherein tracking motion of the first device further includes using dead reckoning to determine the current location of the first device.

9. The method as recited in claim 8, wherein tracking motion of the first device further includes:

tracking location of static features in the 3D space using image information; and

correcting shift in dead reckoning based on the location of the static features.

10. The method as recited in claim 9, wherein the static features are selected from a group consisting of an edge of a table, a corner in a room, a window, or a television set.

11. The method as recited in claim 1, further including:

resetting location tracking modules in the first device and the second device when the reference point is identified.

12. The method as recited in claim 1, wherein the viewing angle is defined as a vector with origin in a center of the display and with a direction perpendicular to and away from the display.

13. A method for generating an interactive space viewable through at least a first and a second device, the method comprising:

detecting a tapping between the first device and the second device, wherein the first device and the second device are handheld devices;

exchanging synchronization information data between the first and second devices to identify a reference point in a three-dimensional (3D) space relative to a physical location of the first and second devices in the 3D space when the tapping was detected; and

generating views of an interactive scene in corresponding displays of the first and second devices, wherein the corresponding device acts as camera viewing into the interactive scene, the interactive scene is tied to the reference point and includes virtual objects, each view showing all or part of the interactive scene and showing the interactive scene as observed from a current location of the corresponding device with a viewing angle of the corresponding camera, wherein device movement in the 3D space causes the corresponding view to change according to the current location and the viewing angle.

14. The method as recited in claim 13, wherein detecting the tapping further includes:

detecting that the first device has tapped a first object at a first time;

receiving information that the second device has tapped a second object at a second time; and

determining that the first device has tapped the second device when the first and the second times are substantially simultaneous.

15. The method as recited in claim 14, wherein detecting that a first device has tapped a first object further includes:

identifying an abrupt change in motion of the first device.

16. The method as recited in claim 15, wherein identifying the abrupt change is based on information from a 3-axis accelerometer.

17. The method as recited in claim 13, wherein each device is configured for interaction with the virtual objects.

18. A portable handheld device for sharing a virtual reality among portable handheld devices, the portable handheld device comprising:

a position module for tracking a position of the portable handheld device, the position module configured to detect a location of a second device in reference to a location of the portable handheld device;

a communications module for exchanging synchronization information data between the portable handheld device and the second device, wherein a reference point in a three-dimensional (3D) space relative to a physical location of the portable handheld device in the 3D space is identified based on the synchronization information;

a view generator that creates a view of an interactive scene that is tied to the reference point and includes virtual objects, wherein the portable handheld device acts as camera viewing into the interactive scene, the view showing all or part of the interactive scene as observed from a current location of the portable handheld device with a viewing angle of the camera, wherein portable handheld device movement in the 3D space causes the view to change according to the current location and the viewing angle; and

a display for showing the view.

19. The portable handheld device of claim 18, wherein the position module includes location tracking modules, wherein the location tracking modules further include:

a 3-axis accelerometer;

a 3 axis gyroscope;

a camera;

a GPS module; and

a depth camera.

20. The portable handheld device of claim 18, wherein detecting the location of the second device further includes:

detecting that the portable handheld device has tapped a first object at a first time;

receiving information that the second device has tapped a second object at a second time; and

determining that the portable handheld device has tapped the second device when the first and the second times are substantially simultaneous.