Systems for using three-dimensional object as controller in an interactive game

• US 10,195,528 B2
• Filed: 12/01/2016
• Issued: 02/05/2019
• Est. Priority Date: 12/17/2007
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Zalewski, Gary, Gillo, Tomas
• Primary Examiner(s): Chan, Allen
• Application Number:
  US15/366,980
• Publication Number:
  US 20170080338A1
• Time in Patent Office: 796 Days
• Field of Search: 463 33
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8011 Ball
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Systems and method are provided for executing an application for interfacing with a virtual reality scene. In one method includes capturing image data of an object using a camera and processing the image data of the object to identify geometric parameters of object. During an interactive session processed by a computer system, the method uses the captured image data and the identified geometric parameters of the object to track movements of the object. The method then causes rendering on a display, the virtual object. The virtual object is used to represent the object. The virtual object rendered on the display such that movement of the object causes rendering of corresponding movement by the virtual object on the display screen.

Claims

1. A system, comprising:

a camera configured to capture data of a three-dimensional space, the three-dimensional space including an object to be used as a controller; and

a computer system being interfaced with the camera and a display screen;

the computer system providing a graphical display that is rendered on the display screen, the graphical display receiving user input in response movements of the object in the three-dimensional space, wherein the object is configured to be tracked using at least image data captured by the camera, wherein the object is rendered in the graphical display as a virtual object for interfacing with interactive content of the graphical display,

wherein the interfacing includes enabling the virtual object to cause one or more actions to be executed by the computer system when the object is moved during an interactive session of interfacing with the interactive content that is executed by the computer system;

wherein the object has a shape that is captured in said image data by the camera.

2. The system as recited in claim 1, wherein the computer system executes one of a program, a game program, an application, a search engine, a communication program, or a combination of two or more thereof.

3. The system as recited in claim 1, wherein said shape is trackable for changes in position when used to provide input to the computer system.

4. The system as recited in claim 1, wherein the camera is configured to capture one or more red-green-blue (RGB) image frames of said image data, and additionally capture one or more of depth data, infrared lighting, or distance data.

5. The system as recited in claim 1, wherein depth data is captured by said camera or another capture device to enable the computer system to construct at least a partial wire frame model of the object for identifying said shape.

6. The system as recited in claim 1, wherein the computer system causes the camera to capture a sequence of images to enable identification of the shape or changes in position of the shape of the object.

7. The system as recited in claim 2, wherein the computer program is configured to execute images to produce a virtual reality space having said interactive content, wherein the object is a real-world object and the real-world object is represented in the virtual reality space as said virtual object, such that movement of the object are shown as movements of the virtual object in the virtual reality space.

8. A computer implemented method, comprising:

capturing image data of an object using a camera;

processing the image data of the object to identify geometric parameters of object;

during an interactive session processed by a computer system, using the captured image data and the identified geometric parameters of the object to track movements of the object; and

rendering on a display, a virtual object, that is used to represent the object, the virtual object being rendered on the display such that movement of the object causes rendering of corresponding movement by the virtual object on a display screen.

9. The computer implemented method of claim 8, wherein the virtual object is rendered in a virtual reality scene rendered on the display.

10. The computer implemented method of claim 8, wherein the camera is one that is configured to capture red-green-blue (RGB) images, or infrared images, or depth images, or a combination to two or more thereof.

11. The computer implemented method of claim 8, wherein the movements of the object are mapped to specific inputs to be made by the virtual object in a virtual reality scene.

12. The computer implemented method of claim 8, further comprising,

triggering activation of actions by the virtual object during the interactive session in response to select movements, or changes in position of the object, as captured by the camera.

13. The computer implemented method of claim 8, wherein actions by virtual objects are pre-correlated to one or more of said select movements or changes in positions or gestures performed with the object, wherein said pre-correlation is for a specific application executed during the interactive session.

14. The computer implemented method of claim 8, wherein movements include one or more of moving the object, turning the object, waving the object, tapping the object, shaking the object, pointing with the object, gesturing with the object, or combination of two or more thereof during the interactive session.

15. The computer implemented method of claim 8, wherein the object is a physical object that is hand-held by a user.

16. The computer implemented method of claim 8, wherein the interactive session is of a video game or an application that renders a virtual reality scene for rendering the virtual object, the virtual reality scene having one or more objects and the virtual object used to interact or interface with the one or more objects.

17. Computer readable media having program instructions for executing an application for interfacing with a virtual reality scene, the computer readable media, comprising:

program instructions for capturing image data of an object using a camera;

program instructions for processing the image data of the object to identify geometric parameters of object;

program instructions for, during an interactive session processed by a computer system, using the captured image data and the identified geometric parameters of the object to track movements of the object; and

program instructions for rendering on a display, the virtual object, that is used to represent the object, the virtual object being rendered on the display such that movement of the object causes rendering of corresponding movement by the virtual object on the display screen.

18. The computer readable media of claim 17, wherein the virtual object is rendered in the virtual reality scene rendered on the display.

19. The computer readable media of claim 17, wherein the camera is one that is configured to capture red-green-blue (RGB) images, or infrared images, or depth images, or a combination to two or more thereof.

20. The computer readable media of claim 17,

wherein the movements of the object are mapped to specific inputs to be made by the virtual object in the virtual reality scene;

wherein movements include one or more of moving the object, turning the object, waving the object, tapping the object, shaking the object, pointing with the object, gesturing with the object, or combination of two or more thereof during the interactive session;

wherein the interactive session is of a video game or the application that renders the virtual reality scene for rendering the virtual object, the virtual reality scene having one or more objects and the virtual object used to interact or interface with the one or more objects.