Methods for Interfacing With an Interactive Application Using a Controller With an Integrated Camera

• US 20160080724A1
• Filed: 11/24/2015
• Published: 03/17/2016
• Est. Priority Date: 03/17/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Zalewski, Gary
• Granted Patent: US 10,129,526 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/90 Constructional details or arrangements of video game devices not provided for in groups A63F13/20 or A63F13/25, e.g. housing, wiring, connections or cabinets
• CPC Class Codes: A63F 2300/1018 Calibration; Key and button assignment
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A computer implemented method for using a controller with at least one integrated camera is provided. The method includes identifying fixed points associated with objects disposed in a three-dimensional space using a depth camera that is part of with a controller. The method includes analyzing the identified fixed points captured by the depth camera and determining changes in position of the controller based on changes in position of the fixed points as determined from depth data captured by the depth camera. The method further includes triggering an input command when the controller is determined to in a particular positon based on the determined changes in position.

Claims

1. A computer implemented method, comprising:

identifying fixed points associated with objects disposed in a three-dimensional space using a depth camera that is part of with a controller;

analyzing the identified fixed points captured by the depth camera;

determining changes in position of the controller based on changes in position of the fixed points as determined from depth data captured by the depth camera; and

triggering an input command when the controller is determined to in a particular positon based on the determined changes in position.

2. The computer implemented method of claim 1, wherein the input command is communicated to an application processed by a computing device, the computing device being in communication with the controller.

3. The computer implemented method of claim 2, wherein the input command is configured to instruct the application to perform an action.

4. The computer implemented method of claim 1, wherein the depth data captured by the depth camera includes a plurality of frames.

5. The computer implemented method of claim 2, wherein a plurality of movements, when performed by the controller, are correlated to control of actions by the application.

6. The computer implemented method of claim 2, wherein the application includes instructions to render a virtual world on a monitor and the actions by the application are correlated to movement within the virtual world.

7. The computer implemented method of claim 2, further comprising,

calibrating the application by capturing a plurality of depth data in a plurality of directions in the three-dimensional space so as to identify the fixed points.

8. The computer implemented method of claim 1, wherein the identified fixed points are associated with real-world objects in the three-dimensional space, and wherein particular ones of the fixed points are grouped and related to ones of the real-world objects.

9. The computer implemented method of claim 1, wherein a group of the fixed points are used to create a virtual world model of at least part of the three-dimensional space, the virtual world model being used to determine ratios of distances between the group of recognized fixed points when the controller is located at various positions within the three-dimensional space.

10. The computer implemented method of claim 1, wherein the controller further includes an image capture device in addition to the depth camera, the image data being processed along with the depth data to define wireframe models using the fixed points of one or more objects in the three-dimensional scene.

11. The computer implemented method of claim 1, wherein the determined changes in position of the controller include spatial position in the three-dimensional scene.

12. The computer implemented method of claim 11, wherein the spatial position includes pitch, yaw and roll parameters of the changes in position of the controller.

13. The computer implemented method of claim 2, further comprising, filtering the fixed points to a set of fixed points that identify corner features of real-world objects in the three-dimensional space.

14. The computer implemented method of claim 13, wherein the filtering further includes using color contrast of the real-world objects in the three-dimensional space to identify the set of fixed points, wherein the color contrast is captured by an image camera integrated with the controller along with the depth camera.

15. Computer readable media having program instructions for processing a computer implemented method, the computer readable media comprising:

program instructions for identifying fixed points associated with objects disposed in a three-dimensional space using a depth camera that is part of with a controller;

program instructions for analyzing the identified fixed points captured by the depth camera;

program instructions for determining changes in position of the controller based on changes in position of the fixed points as determined from depth data captured by the depth camera; and

program instructions for triggering an input command when the controller is determined to in a particular positon based on the determined changes in position;

wherein the input command is communicated to an application processed by a computing device, the computing device being in communication with the controller.

16. The computer readable media of claim 15, wherein the input command is configured to instruct the application to perform an action.

17. The computer readable media of claim 15, wherein the identified fixed points are associated with real-world objects in the three-dimensional space, and wherein particular ones of the fixed points are grouped and related to ones of the real-world objects.

18. The computer readable media of claim 15 wherein a group of the fixed points are used to create a virtual world model of at least part of the three-dimensional space, the virtual world model being used to determine ratios of distances between the group of recognized fixed points when the controller is located at various positions within the three-dimensional space.

19. The computer readable media of claim 15, wherein the controller further includes an image capture device in addition to the depth camera, the image data being processed along with the depth data to define wireframe models using the fixed points of one or more objects in the three-dimensional scene.

20. The computer readable media of claim 15, further comprising,

program instructions for filtering the fixed points to a set of fixed points that identify corner features of real-world objects in the three-dimensional space;

wherein the filtering further includes program instructions for using color contrast of the real-world objects in the three-dimensional space to identify the set of fixed points, wherein the color contrast is captured by an image camera integrated with the controller along with the depth camera.