METHOD AND APPARATUS FOR TRACKING THREE-DIMENSIONAL MOVEMENTS OF AN OBJECT USING A DEPTH SENSING CAMERA

• US 20140078312A1
• Filed: 10/21/2013
• Published: 03/20/2014
• Est. Priority Date: 07/27/2002
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Zalewski, Gary M., Marks, Richard, Mao, Xiadong
• Granted Patent: US 10,220,302 B2
• Time in Patent Office: Days
• US Class Current: 348/169
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/50 Controlling the output signals based on the game progress
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/248 involving reference images or patches

Abstract

A controller (110) for controlling an execution of a game program by a processor for enabling an interactive game to be played by a user includes a body (111) having a section to be oriented towards a screen when a progress of a game provided via execution of the game apparatus is displayed upon the screen, and at least one photonically detectable (“PD”) element (e.g. 122, 124, 126, and/or 128) assembled with the body, a position of the photonically detectable element within an image being recordable by an image capture device (112) when the section is oriented at least partly towards the screen, wherein positions of the PD element at different points in time are quantifiable to quantify movement of the body in space.

Claims

1-20. -20. (canceled)

21. A method for use in obtaining input for an application, comprising:

establishing communications with an image capture device that is capable of providing first output data that includes depth measurements;

using the image capture device to capture images of an object as the object moves in a three-dimensional manner;

receiving the first output data provided by the image capture device as the image capture device captures images of the object;

receiving additional output data from at least one sensor other than the image capture device that can be used to determine motion and/or orientation of the object, wherein the at least one sensor other than the image capture device does not comprise an image capture device; and

tracking the three-dimensional movements of the object by mixing the first output data provided by the image capture device with the additional output data provided by the at least one sensor other than the image capture device;

wherein the image capture device comprises a three-dimensional camera that measures a depth of a pixel and determines a distance of the pixel from the camera.

22. The method of claim 21, further comprising:

changing weights that are applied to the first output data and the additional output data in the mixing.

23. The method of claim 21, wherein the object comprises one or more light sources

24. The method of claim 23, wherein the one or more light sources are arranged in an arcuate pattern.

25. The method of claim 23, wherein the one or more light sources are arranged in a geometric shape.

26. The method of claim 21, wherein the object comprises a controller.

27. The method of claim 21, further comprising:

generating input for the application based on the tracked three-dimensional movements of the object.

28. The method of claim 21, wherein the tracking the three-dimensional movements of the object comprises:

determining position information for the object.

29. The method of claim 21, wherein the tracking the three-dimensional movements of the object comprises:

determining orientation information for the object.

30. The method of claim 21, wherein the tracking the three-dimensional movements of the object comprises:

determining acceleration information for the object.

31. The method of claim 21, wherein the tracking the three-dimensional movements of the object comprises:

determining velocity information for the object.

32. The method of claim 21, wherein the tracking the three-dimensional movements of the object comprises:

determining at least one of a tilt, pitch, yaw, and roll for the object.

33. A computer program product comprising a non-transitory medium for embodying a computer program for input to a computer and a computer program embodied in the non-transitory medium for causing the computer to perform steps comprising:

establishing communications with an image capture device that is capable of providing first output data that includes depth measurements;

using the image capture device to capture images of an object as the object moves in a three-dimensional manner;

receiving the first output data provided by the image capture device as the image capture device captures images of the object;

receiving additional output data from at least one sensor other than the image capture device that can be used to determine motion and/or orientation of the object, wherein the at least one sensor other than the image capture device does not comprise an image capture device; and

tracking the three-dimensional movements of the object by mixing the first output data provided by the image capture device with the additional output data provided by the at least one sensor other than the image capture device;

wherein the image capture device comprises a three-dimensional camera that measures a depth of a pixel and determines a distance of the pixel from the camera.

34. The computer program product of claim 33, wherein the computer program further causes the computer to perform a step comprising:

changing weights that are applied to the first output data and the additional output data in the mixing.

35. The computer program product of claim 33, wherein the object comprises one or more light sources.

36. The computer program product of claim 35, wherein the one or more light sources are arranged in an arcuate pattern.

37. The computer program product of claim 35, wherein the one or more light sources are arranged in a geometric shape.

38. The computer program product of claim 33, wherein the object comprises a controller.

39. The computer program product of claim 33, wherein the computer program further causes the computer to perform a step comprising:

generating input for an application based on the tracked three-dimensional movements of the object.

40. The computer program product of claim 33, wherein the tracking the three-dimensional movements of the object comprises:

determining position information for the object.

41. The computer program product of claim 33, wherein the tracking the three-dimensional movements of the object comprises:

determining orientation information for the object.

42. The computer program product of claim 33, wherein the tracking the three-dimensional movements of the object comprises:

determining acceleration information for the object.

43. The computer program product of claim 33, wherein the tracking the three-dimensional movements of the object comprises:

determining velocity information for the object.

44. The computer program product of claim 33, wherein the tracking the three-dimensional movements of the object comprises:

determining at least one of a tilt, pitch, yaw, and roll for the object.

45. An apparatus, comprising:

an image capture device that is capable of providing first output data that includes depth measurements; and

a processing system that is communicatively coupled to the image capture device, wherein the processing system is configured to execute steps comprising using the image capture device to capture images of an object as the object moves in a three-dimensional manner, receiving the first output data provided by the image capture device as the image capture device captures images of the object, receiving additional output data from at least one sensor other than the image capture device that can be used to determine motion and/or orientation of the object, and tracking the three-dimensional movements of the object by mixing the first output data provided by the image capture device with the additional output data provided by the at least one sensor other than the image capture device;

wherein the at least one sensor other than the image capture device does not comprise an image capture device; and

wherein the image capture device comprises a three-dimensional camera that measures a depth of a pixel and determines a distance of the pixel from the camera.

46. The apparatus of claim 45, wherein the processing system is further configured to execute a step comprising:

changing weights that are applied to the first output data and the additional output data in the mixing.

47. The apparatus of claim 45, wherein the object comprises one or more light sources.

48. The apparatus of claim 47, wherein the one or more light sources are arranged in an arcuate pattern.

49. The apparatus of claim 47, wherein the one or more light sources are arranged in a geometric shape.

50. The apparatus of claim 45, wherein the object comprises a controller.

51. The apparatus of claim 45, wherein the processing system is further configured to execute a step comprising:

generating input for an application based on the tracked three-dimensional movements of the object.

52. The apparatus of claim 45, wherein the tracking the three-dimensional movements of the object comprises:

determining position information for the object.

53. The apparatus of claim 45, wherein the tracking the three-dimensional movements of the object comprises:

determining orientation information for the object.

54. The apparatus of claim 45, wherein the tracking the three-dimensional movements of the object comprises:

determining acceleration information for the object.

55. The apparatus of claim 45, wherein the tracking the three-dimensional movements of the object comprises:

determining velocity information for the object.

56. The apparatus of claim 45, wherein the tracking the three-dimensional movements of the object comprises:

determining at least one of a tilt, pitch, yaw, and roll for the object.