Calibration of portable devices in a shared virtual space

• US 8,717,294 B2
• Filed: 09/03/2013
• Issued: 05/06/2014
• Est. Priority Date: 03/05/2010
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Weising, George, Miller, Thomas
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): LAM, VINH TANG
• Application Number:
  US14/017,208
• Publication Number:
  US 20140002359A1
• Time in Patent Office: 245 Days
• Field of Search: 345156-158, 463 1- 8, 463 30- 42
• US Class Current: 345/158
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2009/2457 Display screens, e.g. monitors, video displays
• CPC Class Codes: A63F 3/02 Chess; Similar board games
• CPC Class Codes: A63F 9/24 Electric games; Games using electronic circuits not otherwise provided for video games A63F13/00; computerized gaming systems G07F17/32
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G09G 2320/0261 : in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 5/08 : Cursor circuits
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors

Abstract

Methods, systems, and computer programs for generating an interactive space, viewable through at least a first and a second handheld devices, are presented. The method includes an operation for taking an image with a camera in the first device. In addition, the method includes an operation for determining a relative position of the second device with reference to the first device, based on image analysis of the taken image to identify a geometry of the second device. Furthermore, the method includes operations for identifying a reference point in a three-dimensional (3D) space based on the relative position, and for generating views of an interactive scene in corresponding displays of the first device and the second device. The interactive scene is tied to the reference point and includes virtual objects, and each view shows all or part of the interactive scene as observed from a current location of the corresponding device.

Claims

1. A method for generating an interactive space viewable through at least a first device and a second device, the method comprising:

taking an image with a camera in the first device, wherein the first device and the second device are handheld devices;

determining a relative position of the second device with reference to the first device based on image analysis of the image to identify a geometry of the second device;

identifying a reference point in a three-dimensional (3D) space based on the relative position; and

generating views of an interactive scene in corresponding displays of the first device and the second device, wherein the interactive scene is tied to the reference point and includes virtual objects, each view showing all or part of the interactive scene as observed from a current location of the corresponding device.

2. The method as recited in claim 1, wherein determining the relative position further includes:

identifying an outline of the second device in the image.

3. The method as recited in claim 2, wherein determining the relative position further includes:

identifying a size of the outline; and

determining distance to the second device based on the size of the outline of the second device.

4. The method as recited in claim 2, wherein determining the relative position further includes:

identifying a tilt angle of the second device in the image.

5. The method as recited in claim 1, further including:

sending synchronization information from the first device to the second device after identifying the reference point.

6. The method as recited in claim 1, further including:

receiving a command to synchronize the first device and the second device before taking the image with a camera.

7. The method as recited in claim 1, wherein generating views of the interactive scene further includes:

generating a first view on a first display in the first device presenting the interactive scene as observed from a point on the first display with a viewing direction perpendicular to the first display, wherein the first view does not depend on an eye position of a user holding the first device.

8. The method as recited in claim 7, wherein movement of the first device in the 3D space causes the corresponding view to change according to the current location.

9. The method as recited in claim 1, wherein generating views of the interactive scene further includes:

generating a first view on a first display in the first device presenting the interactive scene as observed from an eye of a user holding the first device, wherein a first display on the first device acts as a window into the interactive scene, wherein the first view depends on a position of the eye of the user.

10. The method as recited in claim 1, wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

11. A method for generating an interactive space viewable through at least a first device and a second device, the method comprising:

taking an image with a camera in the first device, wherein the first device and the second device are handheld devices;

determining a relative position of the second device with reference to the first device based on image analysis of the image to identify a light source in the second device;

identifying a reference point in a three-dimensional (3D) space based on the relative position; and

generating views of an interactive scene in corresponding displays of the first device and the second device, wherein the interactive scene is tied to the reference point and includes virtual objects, each view showing all or part of the interactive scene as observed from a current location of the corresponding device.

12. The method as recited in claim 11, further including:

detecting, in the second device, a command to synchronize the first device and the second device; and

turning on the light source in the second device in response to the command to synchronize.

13. The method as recited in claim 11, further including:

sending synchronization information from the first device to the second device after identifying the reference point.

14. The method as recited in claim 11, wherein generating views of the interactive scene further includes:

generating a first view on a first display in the first device presenting the interactive scene as observed from a point on the first display with a viewing direction perpendicular to the first display, wherein the first view does not depend on an eye position of a user holding the first device.

15. The method as recited in claim 11, wherein generating views of the interactive scene further includes:

generating a first view on a first display in the first device presenting the interactive scene as observed from an eye of a user holding the first device, wherein a first display on the first device acts as a window into the interactive scene, wherein the first view depends on a position of the eye of the user.

16. A method for generating an interactive space viewable through at least a first device and a second device, the method comprising:

taking an image with a camera in the first device, wherein the first device and the second device are handheld devices;

determining a relative position of the second device with reference to the first device based on image analysis of the image to identify a graphic shown on a second display of the second device;

identifying a reference point in a three-dimensional (3D) space based on the relative position; and

generating views of an interactive scene in corresponding displays of the first device and the second device, wherein the interactive scene is tied to the reference point and includes virtual objects, each view showing all or part of the interactive scene as observed from a current location of the corresponding device.

17. The method as recited in claim 16, further including:

prompting a user holding the second device, before taking the image, to place the second display in a position facing the camera in the first device.

18. The method as recited in claim 16, wherein the graphic shown on the second display includes a brightly lit second display.

19. The method as recited in claim 16, wherein the graphic shown on the second display includes a geometric figure.

20. The method as recited in claim 16, wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.