DETERMINATION OF CONTROLLER THREE-DIMENSIONAL LOCATION USING IMAGE ANALYSIS AND ULTRASONIC COMMUNICATION

• US 20080261693A1
• Filed: 06/24/2008
• Published: 10/23/2008
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America (Sony Group Corp.)
• Inventors: Zalewski, Gary M.
• Granted Patent: US 8,323,106 B2
• Time in Patent Office: Days
• US Class Current: 463/31
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 13/843 involving concurrently two or more players on the same game device, e.g. requiring the use of a plurality of controllers or of a specific view of game data for each player
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Game interface tracks the position of one or more game controllers in 3-dimensional space using hybrid video capture and ultrasonic tracking system. The captured video information is used to identify a horizontal and vertical position for each controller within a capture area. The ultrasonic tracking system analyzes sound communications to determine the distances between the game system and each controller and to determine the distances among the controllers. The distances are then analyzed by the game interface to calculate the depths within the capture area for each controller.

Claims

1. A method for establishing communication between a computer program in a computing system and a controller held in a hand of a user, comprising:

capturing image data in a capture area in front of a display, the capturing configured to identify a two-dimensional location of the controller in the capture area; and

capturing one-way sound communication between a location proximate to the display and a location at the controller, the sound communication being used to compute a third-dimensional location of the controller in the capture area.

2. The method as recited in claim 1, wherein identifying a two-dimensional location of the controller further includes detecting a spherical shape within the image data, the spherical shape being associated with the controller.

3. The method as recited in claim 1, wherein the captured image data corresponds to one of either an red-green-blue (RGB) pixel image data, a black and white pixel image data, or infrared data.

4. The method as recited in claim 1 further including, synchronizing clocks between two ends of the one-way sound communication before capturing the one-way sound communication.

5. The method as recited in claim 1, wherein the one-way sound communication includes a sender of the sound communication at the controller.

6. The method as recited in claim 1, wherein the one-way sound communication includes a sender of the sound communication at the location proximate to the display.

7. The method as recited in claim 1, further including,

capturing one-way sound communication between a location proximate to the display and a location at a second controller, and

bi-directional sound communication between the controller and the second controller.

8. The method as recited in claim 1, wherein computing a third-dimensional location further includes,

calculating a distance from the location proximate to the display and the location at the controller based on a time travel of the sound in the one-way sound communication, and

calculating a depth within the capture area based on the distance,

wherein the depth is an orthogonal measure to both dimensions of the two-dimensional location.

9. The method as recited in claim 1, further including,

presenting in a graphical user interface in the display an object based on the two-dimensional location and the third-dimensional location.

10. A system for establishing communication between a computer program and a controller, comprising:

an image capture device for capturing image data in a capture area in front of a display;

a sound capture device for capturing sound data in the capture area; and

a computing system for executing the computer program, the computing system being connected to the display, the image capture device, and the sound capture device;

the controller to be held by a user with a single hand, the controller including,

a body with a first end and a second end;

a grip area at about the first end;

an input pad defined between the first end and the second end for entering input data to be transmitted to the computing system;

a spherical-type section defined at the second end for facilitating image recognition by the image capture device; and

a sound emitting device near the second end, the sound emitting device directed away from the user when held and configured for one-way sound communication with the sound capture device;

wherein the computer program identifies a two-dimensional location of the controller based on the captured image data and a third-dimensional location of the controller based on the captured sound data.

11. The system as recited in claim 10, wherein the spherical-type section further includes a retro-reflective area.

12. The system as recited in claim 10, wherein the spherical-type section has a predetermined color to assist in identifying the spherical-type section location in the captured image data.

13. The system as recited in claim 10, wherein the controller further includes,

a light source inside the spherical-type section,

wherein the spherical-type section is translucent,

wherein the light source illuminates the spherical-type section to assist in identifying the spherical-type section location in the captured image data.

14. The system as recited in claim 13, wherein the light source includes an infrared light source, wherein the image capture device further includes an infrared camera.

15. The system as recited in claim 14, wherein the image capture device further includes a visible-spectrum camera, wherein the image data from the infrared camera and the visible-spectrum camera are combined in the identification of the two-dimensional location.

16. The system as recited in claim 10 further comprising,

a microphone in the controller,

a second controller configured like the controller,

wherein a three-dimensional location of the second controller is identified by the computer program,

wherein the controller and the second controller are configured for two-way sound communication used to determine a distance between the controller and the second controller, the distance being transmitted to the computing system,

wherein the computer program uses the determined distance between controllers in the identification of the three-dimensional locations of the controllers.

17. The system as recited in claim 16, wherein the controller and the second controller are physically connected serially along a longitudinal dimension,

wherein both controllers are tracked by the computer program and the identified locations of the controllers are combined to improve accuracy.

18. The system as recited in claim 17, further comprising:

an adapter defined between ends of the controllers to enable connection.

19. The system as recited in claim 16, wherein the controller and the second controller are physically connected in parallel along a longitudinal dimension, wherein a plate is used to connect the controllers to form an integrated controller design to be held with two hands.

20. The system as recited in claim 10 further including,

a speaker connected to the computer system,

wherein the controller further includes a controller sound capturing device,

wherein the controller and the computer system establish an inverse one-way sound communication via the speaker and the controller sound capturing device.

21. The system as recited in claim 10, wherein the identification of the third-dimensional location further includes,

calculating a depth value of the third-dimensional location by measuring time of flight and phase coherence of the one-way sound communication.

22. The system as recited in claim 10, wherein the input pad is removable in order to accommodate different button configurations for the input pad.

23. The system as recited in claim 10, wherein the computer program tracks user movement by detecting a position of a user head within the captured image data or a position of a user torso within the captured image data.

24. The system as recited in claim 10, wherein the controller further includes an inertial analyzer for tracking inertial activity of the controller, the inertial activity transferred to the computer program in impact an action.

25. The system as recited in claim 10, wherein the sound emitting device further includes an acoustical chamber inside the controller, the acoustical chamber defined to direct and receive sound in one or more directions.

26. The system as recited in claim 10, wherein the computer program samples some of the captured image and sound data, the computer program combining the sampled data during the identification of the location of the controller.

27. A system for establishing communication between a computer program and a controller, comprising:

a first set and a second set of light emitters facing a capture area in front of a display, the first and second set of light emitters located near the display;

a sound capture device for capturing sound data in the capture area; and

a computing system for executing the computer program, the computing system being connected to the display, and the sound capture device;

the controller to be held by a user with a single hand, the controller including,

a body with a first end and a second end;

a grip area at about the first end;

an input pad defined between the first end and the second end for entering input data to be transmitted to the computing system;

an image capture device near the second end for capturing image data of an area where the first and second set of light emitters are located; and

a sound emitting device near the second end, the sound emitting device directed away from the user when held and configured for one-way sound communication with the sound capture device;

wherein the computer program identifies a two-dimensional location of the controller based on the captured image data and a third-dimensional location of the controller based on the captured sound data.

28. The system as recited in claim 27, wherein the controller transmits the captured image data to the computing system.

29. The system as recited in claim 27, wherein a processor in the controller calculates a location of the controller based on the captured image data, wherein the controller transmits the calculated location to the computing system.

30. A computer program embedded in a computer-readable storage medium, when executed by one or more processors, for establishing communication between a computer program in a computing system and a controller held in a hand of a user, the computer program comprising:

program instructions for capturing image data in a capture area in front of a display, the capturing configured to identify a two-dimensional location of the controller in the capture area; and

program instructions for capturing one-way sound communication between a location proximate to the display and a location at the controller, the sound communication being used to compute a third-dimensional location of the controller in the capture area.