Determination of controller three-dimensional location using image analysis and ultrasonic communication

• US 8,784,203 B2
• Filed: 09/13/2012
• Issued: 07/22/2014
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Zalewski, Gary M.
• Primary Examiner(s): Lewis, David L
• Assistant Examiner(s): HALL, SHAUNA-KAY N
• Application Number:
  US13/615,355
• Publication Number:
  US 20130002550A1
• Time in Patent Office: 677 Days
• US Class Current: 463/31
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 13/843 involving concurrently two or more players on the same game device, e.g. requiring the use of a plurality of controllers or of a specific view of game data for each player
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Methods, systems, and computer programs are presented for determining the location of a controller. One method includes an operation for capturing image data of a capture area in front of a display. Additionally, the method includes another operation for capturing sound data emitted by the controller in the capture area in front of the display. The two-dimensional location of the controller is calculated based on the captured image data, and the third dimensional location of the controller is calculated based on the captured sound data.

Claims

1. A method for determining location of a controller, the method comprising:

capturing image data of a capture area in front of a display;

capturing sound data emitted by the controller in the capture area in front of the display;

calculating a two-dimensional location of the controller based on the captured image data; and

calculating a third dimensional location of the controller based on the captured sound data, wherein calculating the third dimensional location further includes;

measuring a time of flight of the captured sound data; and

calculating a depth within the capture area based on the time of flight of the captured sound data, wherein the depth is an orthogonal measure to both dimensions of the two-dimensional location, wherein operations of the method are executed by a processor.

2. The method as recited in claim 1, wherein calculating the two-dimensional location of the controller further includes:

detecting a spherical shape within the image data, the spherical shape being associated with a spherical-type section in the controller.

3. The method as recited in claim 2, wherein the spherical-type section is operable to emit one of a plurality of light colors to assist in identifying the spherical shape in the captured image data.

4. The method as recited in claim 3, further including:

sending from a processor to the controller a choice of light color from the plurality of light colors, wherein the processor calculates the two-dimensional location, wherein the choice of light color is utilized to identify the spherical shape in the captured image data.

5. The method as recited in claim 1, wherein calculating the third dimensional location further includes:

measuring and a phase coherence of the captured sound data; and

calculating the depth within the capture area based on the time of flight and the phase coherence of the captured sound data.

6. The method as recited in claim 1, further including,

presenting in a graphical user interface in the display an object based on the two-dimensional location and the third dimensional location.

7. The method as recited in claim 1, wherein the sound data is captured with a sound capture device, the method further including:

establishing ultrasound communication between the sound capture device and the controller.

8. The method as recited in claim 1, further including:

detecting a position of a user head within the captured image data or a position of a user torso within the captured image data.

9. The method as recited in claim 1, wherein operations of the method are performed by a computer program when executed by one or more processors, the computer program being embedded in a non-transitory computer-readable storage medium.

10. A system for determining location of a controller, comprising:

an image capture device operable to capture image data of a capture area in front of a display;

a sound capture device operable to capture sound data emitted by the controller in the capture area in front of the display; and

a processor in communication with the image capture device and the sound capture device, wherein the processor is operable to measure a time of flight of the captured sound data, wherein the processor is operable to calculate a two-dimensional location of the controller based on the captured image data, and the processor is operable to calculate a third dimensional location of the controller based on the captured sound data and the time of flight, wherein the third dimensional location is orthogonal to both dimensions of the two-dimensional location.

11. The system as recited in claim 10, wherein the controller includes:

an elongated body with a first end and a second end;

a grip area at about the first end;

an input pad defined between the first end and the second end;

a spherical-type section defined at the second end; and

a sound emitting device.

12. The system as recited in claim 11, wherein the controller further includes an acoustical chamber, the acoustical chamber defined to direct and receive sound in one or more directions.

13. The system as recited in claim 12, wherein the controller further includes an inertial analyzer for tracking inertial activity of the controller, wherein the controller is operable to transmit the inertial activity to the system, wherein the processor is further operable to include the inertial activity when calculating the two-dimensional location and the third dimensional location of the controller.

14. The system as recited in claim 10, wherein the image capture device includes a visible-spectrum camera, wherein the image capture device further includes an infrared camera, wherein the image data from the infrared camera and the visible-spectrum camera are combined in the calculation of the two-dimensional location.

15. The system as recited in claim 10 further including:

a speaker connected to the processor, wherein the controller further includes a controller sound capturing device, wherein the controller and the system are operable to establish sound communication via the speaker, the sound capture device, and the controller sound capturing device.

16. The system as recited in claim 10, wherein the image data corresponds to one of either an red-green-blue (RGB) pixel image data, a black and white pixel image data, or infrared data.

17. The system as recited in claim 10, wherein the processor is operable to measure a phase coherence of the captured sound data, and the processor is operable to calculate the third dimensional location of the controller within the capture area based on the time of flight and the phase coherence of the captured sound data.

18. A system comprising:

a controller including an image capture device;

one or more light emitters facing a capture area in front of a display, the light emitters located near the display;

a sound capture device operable to capture sound data emitted by the controller when the controller is in the capture area in front of the display, and

a processor in communication with the controller, the one or more light emitters and the sound capture device, wherein the processor is operable to receive from the controller image data captured with the image capture device, wherein the processor determines a position of the light emitters within the image captured data and calculates calculate a two-dimensional location of the controller based on the determined position, wherein the processor calculates a third dimensional location of the controller based on the captured sound data, wherein the processor is operable to measure a time of flight of the captured sound data, and the processor is operable to calculate a depth within the capture area based on the time of flight of the captured sound data.

19. The system as recited in claim 18, wherein the processor is operable to measure a phase coherence of the captured sound data, and the processor is operable to calculate the depth within the capture area based on the time of flight and the phase coherence of the captured sound data.