Prop input device and method for mapping an object from a two-dimensional camera image to a three-dimensional space for controlling action in a game program

• US 6,795,068 B1
• Filed: 07/21/2000
• Issued: 09/21/2004
• Est. Priority Date: 07/21/2000
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard L.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): MCCARTNEY, LINZY T
• Application Number:
  US09/621,578
• Time in Patent Office: 1,523 Days
• Field of Search: 382/102, 382/103, 382/164, 463/47.1, 463/3, 463/4, 345/156, 345/700, 345/773, 345/863, 345/419
• US Class Current: 345/419
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

A hand-manipulated prop is picked-up via a single video camera, and the camera image is analyzed to isolate the part of the image pertaining to the object for mapping the position and orientation of the object into a three-dimensional space, wherein the three-dimensional description of the object is stored in memory and used for controlling action in a game program, such as rendering of a corresponding virtual object in a scene of a video display. Algorithms for deriving the three-dimensional descriptions for various props employ geometry processing, including area statistics, edge detection and/or color transition localization, to find the position and orientation of the prop from two-dimensional pixel data.

Claims

1. A method for controlling a game program comprising the steps of:

manipulating a physical object within a field of view of a single video camera connected to a processing unit having a memory area and a display wherein said physical object comprises a cylinder having a solid color;

supplying two-dimensional pixel information picked up by said video camera to said processing unit;

extracting at least one pixel group from said two-dimensional pixel information which corresponds to the physical object;

determining a set of parameters from said at least one pixel group which define a two-dimensional geometric shape wherein said geometric shape defined by said extracted pixel group is a four-sided polygon;

executing a predetermined algorithm on said set of parameters to map the definition of said two-dimensional geometric shape to a three-dimensional definition of a geometric volume, thereby providing a description of said geometric volume in three dimensions which corresponds to the position and orientation of the said manipulated physical object wherein said predetermined algorithm comprises the steps of;

determining a length l of said pixel group;

determining at least two widths w₁ and w₂ of said pixel group at different lengths along said pixel group;

determining an inclination θ

of the object in the X-Y plane of the image between the Y axis and said length l;

determining an inclination Φ

of the object in the Y-Z plane based on a ratio w₁;

w₂ of said two widths; and

determining a depth Z of the object based on a phi-weighted length l_φ

of the pixel group or a phi-weighted number N_φ

of the total number of pixels making up the pixel group;

storing said three-dimensional definition in said memory; and

using said description of said geometric volume to control action shown on said display.

2. A method for controlling a game program comprising the steps of:

manipulating a physical object within a field of view of a single video camera connected to a processing unit having a memory area and a display wherein said physical object comprises a cylinder having a solid color;

supplying two-dimensional pixel information picked up by said video camera to said processing unit;

extracting at least one pixel group from said two-dimensional pixel information which corresponds to the physical object;

determining a set of parameters from said at least one pixel group which define a two-dimensional geometric shape wherein the geometric shape defined by said extracted pixel group is a four-sided polygon;

executing a predetermined algorithm on said set of parameters to map the definition of said two-dimensional geometric shape to a three-dimensional definition of a geometric volume, thereby providing a description of said geometric volume in three dimensions which corresponds to the position and orientation of the manipulated physical object wherein said predetermined algorithm comprises the steps of;

determining a length l of said pixel group;

determining at least two widths w₁ and w₂ of said pixel group at different lengths along said pixel group;

determining an inclination θ

of the object in the X-Y plane of the image between the Y axis and said length l;

determining a depth Z of the object based on an average width w_avg of the pixel group; and

determining an inclination Φ

of the object in the Y-Z plane based on the ratio l;

w_avg of said length l of the pixel group to the average width w_avg;

storing said three-dimensional definition in said memory; and

using said description of said geometric volume to control action shown on said display.

3. A method for controlling a game program comprising the steps of:

manipulating a physical object within a field of view of a single video camera connected to a processing unit having a memory area and a display wherein the physical object further comprises a sphere having one solid color mounted onto an end of a cylinder having another solid color;

supplying two-dimensional pixel information picked up by said video camera to said processing unit;

extracting at least one pixel group from said two-dimensional pixel information which corresponds to the physical object;

determining a set of parameters from said at least one pixel group which define a two-dimensional geometric shape wherein the geometric shape defined by the extracted pixel group for said sphere is a circle, and the geometric shape defined by said extracted pixel group for said cylinder is a four-sided polygon;

executing a predetermined algorithm on said set of parameters to map the definition of said two-dimensional geometric shape to a three-dimensional definition of a geometric volume, thereby providing a description of said geometric volume in three dimensions which corresponds to the position and orientation of the manipulated physical object wherein said predetermined algorithm comprises the steps of;

determining a center point p_s and a total pixel area of the extracted pixel group for said circle;

determining another point p of a center line of the extracted pixel group for said four-sided polygon;

determining a length l between said points p_s and p;

determining an inclination θ

of the object in the X-Y plane of the image between the Y axis and said length l;

determining a depth Z of the object based on a total number N of pixels in the pixel group for said circle; and

determining an inclination Φ

of the object in the Y-Z plane based on said length l between said points p_s and p and said depth Z;

storing said three-dimensional definition in said memory; and

using said description of said geometric volume to control action shown on said display.

4. A method for controlling a game program comprising the steps of:

manipulating a physical object within a field of view of a single video camera connected to a processing unit having a memory area and a display wherein said physical object comprises a cylinder having one solid color, said cylinder having a plurality of circumferential stripes thereon, said stripes being of a different solid color than said one solid color;

supplying two-dimensional pixel information picked up by said video camera to said processing unit;

extracting at least one pixel group from said two-dimensional pixel information which corresponds to the physical object wherein said extracting step further comprises determining a plurality a color transitions between the cylinder and said stripes;

determining a set of parameters from said at least one pixel group which define a two-dimensional geometric shape;

executing a predetermined algorithm on said set of parameters to map the definition of said two-dimensional geometric shape to a three-dimensional definition of a geometric volume, thereby providing a description of said geometric volume in three dimensions which corresponds to the position and orientation of the manipulated physical object wherein said predetermined algorithm comprises the steps of;

setting a threshold value D_t defining a maximum separation in a two-dimensional chrominance color space;

detecting color transitions based on a value D determined from combined chrominance signals Cr and Cb for both of the colors of said stripes and of said cylinder;

determining that a color transition of sufficient magnitude has occurred when said value D exceeds said predetermined threshold;

defining, by said color transitions of sufficient magnitude, respective lengths l₁ and l₂ between said stripes;

determining an inclination Φ

of the object in the Y-Z plane based on a ratio l₁;

l₂ of the lengths l₁ and l₂ between said stripes; and

determining a depth Z of the object based on the sum of said respective lengths and said inclination Φ

of the object in the Y-Z plane;

storing said three-dimensional definition in said memory; and

using said description of said geometric volume to control action shown on said display.

5. A method for controlling a game program comprising the steps of:

manipulating a physical object within a field of view of a single video camera connected to a processing unit having a memory area and a display wherein said physical object comprises a cylinder having one solid color, said cylinder having a helical stripe thereon, said helical stripe being of a different solid color than said one solid color;

supplying two-dimensional pixel information picked up by said video camera to said processing unit;

extracting at least one pixel group from said two-dimensional pixel information which corresponds to the physical object wherein said extracting step further comprises extracting at least one pixel group corresponding to said helical stripe;

determining a set of parameters from said at least one pixel group which define a two-dimensional geometric shape;

executing a predetermined algorithm on said set of parameters to map the definition of said two-dimensional geometric shape to a three-dimensional definition of a geometric volume, thereby providing a description of said geometric volume in three dimensions which corresponds to the position and orientation of the manipulated physical object;

executing a predetermined algorithm for determining a degree of rotation of said object;

storing said three-dimensional definition in said memory; and

using said description of said geometric volume to control action shown on said display.

6. The method according to claim 5, wherein said predetermined algorithm for determining a degree of rotation comprises the steps of:

extracting a pixel group corresponding to said cylinder;

defining a helix H from the pixel group corresponding to said helical stripe;

determining a center line of the pixel group corresponding to said cylinder; and

determining a height h between an end of the extracted pixel group corresponding to said cylinder and a point on said helix H where the center line intersects said helix H, wherein the degree of rotation of said cylinder is determined based on a ratio of said height h to a total length l of the pixel group corresponding to said cylinder.