Dynamic three-dimensional object mapping for user-defined control device

• US 8,542,907 B2
• Filed: 12/15/2008
• Issued: 09/24/2013
• Est. Priority Date: 12/17/2007
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.), Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Zalewski, Gary, Gillo, Tomas
• Primary Examiner(s): JOHNS, ANDREW W
• Application Number:
  US12/335,505
• Publication Number:
  US 20090158220A1
• Time in Patent Office: 1,744 Days
• Field of Search: 382/100, 382/107, 382/154, 382/286, 715/863, 345/156, 345/157, 345/158
• US Class Current: 382/154
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8011 Ball
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A computer-implemented method is provided to interactively capture and utilize a three-dimensional object as a controlling device for a computer system. One operation of the method is capturing depth data of the three-dimensional object. In another operation, the depth data of the three-dimensional object undergoes processing to create geometric defining parameters for the three-dimensional object. The method can also include defining correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system. The method also includes an operation to save the geometric defining parameters of the three-dimensional object to a recognized object database. In another operation, the correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system in response to recognizing the particular actions are also saved to the recognized object database.

Claims

1. A computer implemented method to interactively capture and utilize a three-dimensional object to be a controlling device for a computer system, comprising:

(a) capturing depth data of the three-dimensional object;

(b) processing captured depth data of the three-dimensional object to create geometric defining parameters for the three-dimensional object;

(c) defining correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system; and

(d) saving to a recognized object database the geometric defining parameters of the three-dimensional object and correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system in response to recognizing the particular actions;

(e) activating an interactive session processed by the computer system, the interactive session including;

(i) capturing depth data and analyzing the depth data, the depth data being compared to the geometric defining parameters stored in the recognized object database; and

(ii) rendering on a display, a predefined virtual object, that is selected for the geometric defining parameters found in the analyzed depth data, the predefined virtual object being rendered on the display in place of the three-dimensional object, the predefined virtual object being different than the three dimensional object.

2. The computer implemented method as recited in claim 1, further comprising:

triggering activation of actions during the interactive session in response to actions performed with the three-dimensional object, as defined by the correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the predefined virtual object on the display.

3. The computer implemented method as in claim 2, wherein the actions performed with the three-dimensional object include moving the object, turning the object, waving the object, tapping the object, shaking the object, pointing with the objects or combinations thereof.

4. The computer implemented method as recited in claim 1, wherein depth data for the three-dimensional object is captured from a plurality of angles, the plurality of angles allowing the computer system to process the depth data and create the geometric parameters for the three-dimensional object.

5. The computer implemented method as recited in claim 1, wherein the correlations are defined by assigning a particular action performed with the three-dimensional object to a pre-defined particular action to be performed by the computer system.

6. The computer implemented method as recited in claim 1, wherein defining the correlations by having the user interactively move the three-dimensional object in response to a prompt for a pre-defined particular action.

7. The computer implemented method as recited in claim 1, creating macro correlations for a remote control so once a new three-dimensional object is defined, basic movements have the same effect.

8. The computer implemented method as recited in claim 1, saving multiple three-dimensional objects in the recognized object database and switching between them in real-time, each object having customized controls and a corresponding virtual object that is to be displayed on the display in place of the captured depth data of the three-dimensional object.

9. A computer implemented method to interactively capture and utilize a three-dimensional object to be a controlling device for a computer system, comprising:

(a) identifying the three-dimensional object, the identifying including,

(i) capturing depth data of the three-dimensional object;

(ii) processing captured depth data of the three-dimensional object to create geometric defining parameters for the three-dimensional object;

(iii) defining correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system; and

(iv) saving to a recognized object database the geometric defining parameters of the three-dimensional object and correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system; and

(b) presenting the three-dimensional object to a camera;

(c) displaying a virtual object on a display in place of the three-dimensional object, the virtual object being different than the three dimensional object; and

(d) moving the presented three-dimensional object in front of the camera so as to trigger one or more of the particular actions to be performed by the computer system and displayed as actions by the virtual object on the display.

10. The computer implemented method as in claim 9, further comprising:

(e) activating an interactive session processed by the computer system, the interactive session includes processing through (b) and (c) at particular times during the interactive session.

11. The computer implemented method as in claim 10, wherein during the interactive session of (e), the method includes,

(i) capturing depth data using the camera and analyzing the depth data, the depth data being compared to the geometric defining parameters stored in the recognized object database;

and

(ii) triggering activation of actions during the interactive session in response to actions performed with the three-dimensional object, as defined by the correlations between particular actions performed with the three-dimensional object and particular actions to be performed by the computer system.

12. The computer implemented method as recited in claim 9, wherein depth data for the three-dimensional object is captured from a plurality of angles by the camera that is a single camera, the plurality of angles allowing the computer system to process the depth data and create the geometric parameters for the three-dimensional object.

13. The computer implemented method as recited in claim 9, wherein the correlations are defined by assigning a particular action performed with the three-dimensional object to a pre-defined particular action to be performed by the computer system.

14. The computer implemented method as recited in claim 9, wherein defining the correlations by having the user interactively move the three-dimensional object in response to a prompt for a pre-defined particular action.

15. The computer implemented method as recited in claim 9, creating macro correlations for a remote control so once a new three-dimensional object is defined, basic movements have a same effect.

16. The computer implemented method as recited in claim 9, saving multiple three-dimensional objects in the recognized object database and switching between them in real-time, each object having customized controls and a corresponding virtual object that is to be displayed on the display in place of the captured depth data of the three-dimensional object.

17. The computer implemented method as in claim 9, wherein the actions performed with the three-dimensional object include moving the object, turning the object, waving the object, tapping the object, shaking the object, pointing with the objects or combinations thereof.