System and method for permitting three-dimensional navigation through a virtual reality environment using camera-based gesture inputs

• US 6,195,104 B1
• Filed: 12/23/1997
• Issued: 02/27/2001
• Est. Priority Date: 12/23/1997
• Status: Expired due to Fees

• Current Assignee: Philips Electronics North America Corporation (Koninklijke Philips N.V.)
• Original Assignee: Philips Electronics North America Corporation (Koninklijke Philips N.V.)
• Inventors: Lyons, Damian M.
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US08/996,677
• Time in Patent Office: 1,162 Days
• Field of Search: 345/419, 345/420, 345/427, 345/433, 345/473, 345/475, 345/474
• US Class Current: 345/473
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

A system and method for constructing three-dimensional images using camera-based gesture inputs of a system user. The system comprises a computer-readable memory, a video camera for generating video signals indicative of the gestures of the system user and an interaction area surrounding the system user, and a video image display. The video image display is positioned in front of the system users. The system further comprises a microprocessor for processing the video signals, in accordance with a program stored in the computer-readable memory, to determine the three-dimensional positions of the body and principle body parts of the system user. The microprocessor constructs three-dimensional images of the system user and interaction area on the video image display based upon the three-dimensional positions of the body and principle body parts of the system user. The video image display shows three-dimensional graphical objects superimposed to appear as if they occupy the interaction area, and movement by the system user causes apparent movement of the superimposed, three-dimensional objects displayed on the video image display.

Claims

1. A system for constructing three-dimensional images using camera-based gesture inputs of a user of the system, comprising:

a computer-readable memory means;

means for receiving video signals indicative of the gestures of the system user and an interaction area surrounding the system user;

means for displaying video images, the video image display means being positioned in front of the system user; and

means for processing the video signals, in accordance with a program stored in the computer-readable memory means, to determine the three-dimensional positions of the body and principle body parts of the system user, wherein the video signal processing means constructs three-dimensional images of the system user and interaction area on the video image display means based upon the received video signals, the video image display means displays three-dimensional graphical objects superimposed to appear as if they occupy the interaction area, and movement by the system user causes apparent movement of the superimposed, threedimensional objects displayed on the video image display means.

2. A system for constructing three-dimensional images using camera-based gesture inputs of a user as recited in claim 1, wherein the superimposed, three-dimensional objects appear to move as if they were physical objects moving in the interaction area.

3. A system for constructing three-dimensional images using camera-based gesture inputs of a user as recited in claim 1, wherein the video signal processing means constructs the three-dimensional images of the system user by:

projecting two-dimensional positions (u, v) of the feet of the system user to three-dimensional coordinates (x, y, z) of the feet;

mapping the head and hands of the system user to three-dimensional coordinates assuming the head and hands are offset from a z position of the feet;

using a height (h) of the system user with biometric data to calculate the shoulder offset of the system user from the head, and to calculate the arm length of the system user;

calculating the offset of each arm of the system user from a corresponding foot of the system user; and

supplying the calculated three-dimensional positions of the head, hands and feet of the system user to the video image display means.

4. A system for constructing three-dimensional images using camera-based gesture inputs of a user as recited in claim 3, wherein the superimposed, three-dimensional graphical objects comprise:

a soccer ball having a set position (f); and

a goal area having a set position on video image display means, wherein video signal processing means compares the foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user, moves the soccer ball according to the calculated foot velocity (fv), slows the soccer ball down by a predetermined velocity, and sounds a bell if the soccer ball enters the goal area.

5. A system for constructing three-dimensional images using camera-based gesture inputs of a user as recited in claim 1, wherein the three-dimensional graphical objects comprise:

a soccer ball having a set position (f); and

a goal area having a set position on video image display means, wherein video signal processing means compares a foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user, moves the soccer ball according to the calculated foot velocity (fv), slows the soccer ball down by a predetermined velocity, and sounds a bell if the soccer ball enters the goal area.

6. A method for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system having a computer-readable memory and video image display connected to a microprocessor using a program stored in the computer-readable memory, the method comprising the steps of:

receiving video signals indicative of the gestures of the system user and an interaction area surrounding the system user;

processing the video signals in the microprocessor to determine the three-dimensional positions of the body and principle body parts of the system user;

using the microprocessor to construct three-dimensional images of the system user and interaction area on the video image display based upon the received video signals; and

utilizing the microprocessor to display on the video image display three-dimensional graphical objects superimposed to appear as if they occupied the interaction area, wherein movement by the system user causes apparent movement by the superimposed, three-dimensional objects displayed on the video image display.

7. A method for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system, as recited in claim 6, wherein the superimposed, three-dimensional objects appear to move as if they were physical objects.

8. A method for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system, as recited in claim 6, wherein the three-dimensional image construction step comprises the steps of:

projecting two-dimensional positions (u, v) of the feet of the system user to three-dimensional coordinates (x, y, z) of the feet;

mapping the head and hands of the system user to three-dimensional coordinates assuming the head and hands are offset from a z position of the feet;

using a height (h) of the system user with biometric data to calculate the shoulder offset of the system user from the head, and to calculate the arm length of the system user;

calculating the offset of each arm of the system user from a corresponding foot of the system user; and

supplying the calculated three-dimensional positions of the head, hands and feet of the system user to the video image display.

9. A method for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system, as recited in claim 8, wherein the superimposed, three-dimensional graphical objects comprise a soccer ball having a set position (f), and a goal area having a set position on video image display means, the method further comprising the steps of:

comparing a foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user;

moving the soccer ball according to the calculated foot velocity (fv);

slowing the soccer ball down by a predetermined velocity; and

sounding a bell if the soccer ball enters the goal area.

10. A method for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system, as recited in claim 6, wherein the three-dimensional graphical objects comprise a soccer ball having a set position (f), and a goal area having a set position on the video image display, the method further comprising the steps of:

comparing a foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user;

moving the soccer ball according to the calculated foot velocity (fv);

slowing the soccer ball down by a predetermined velocity; and

sounding a bell if the soccer ball enters the goal area.

11. A computer-readable memory device for storing a program that constructs three-dimensional images using camera-based gesture inputs of a user of a computer system having a video image display connected to a microprocessor using instructions stored in the computer-readable memory device, the computer-readable memory device comprising:

instructions for receiving video signals indicative of gestures of the system user to determine the three-dimensional positions of the body and principle body parts of the system user;

instructions for constructing three-dimensional images of the system user and interaction area on the video image display based upon the received video signals; and

instructions for displaying, on the video image display, three-dimensional graphical objects superimposed to appear as if they occupied the interaction area, wherein movement by the system user causes apparent movement by the superimposed, three-dimensional objects displayed on the video image display.

12. A computer-readable memory device for storing a program that constructs three-dimensional images using camera-based gesture inputs of a user, as recited in claim 11, wherein the superimposed, three-dimensional objects appear to move as if they were physical objects.

13. A computer-readable memory device for storing a program that constructs three-dimensional images using camera-based gesture inputs of a user, as recited in claim 11, wherein the instructions for constructing the three-dimensional image comprise:

instructions for projecting two-dimensional positions (u, v) of the feet of the system user to three-dimensional coordinates (x, y, z) of the feet;

instructions for mapping the head and hands of the system user to three-dimensional coordinates assuming the head and hands are offset from a z position of the feet;

instructions for using a height (h) of the system user with biometric data to calculate the shoulder offset of the system user from the head, and to calculate the arm length of the system user;

instructions for calculating the offset of each arm of the system user from a corresponding foot of the system user; and

instructions for supplying the calculated three-dimensional positions of the head, hands and feet of the system user to the video image display.

14. A computer-readable memory device for storing a program that constructs three-dimensional images using camera-based gesture inputs of a user, as recited in claim 13, wherein the three-dimensional graphical objects comprise a soccer ball having a set position (f), and a goal area having a set position on the video image display, the computer-readable memory device further comprising:

instructions for comparing a foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user;

instructions for moving the soccer ball according to the calculated foot velocity (fv);

instructions for slowing the soccer ball down by a predetermined velocity; and

instructions for sounding a bell if the soccer ball enters the goal area.

15. A computer-readable memory device for storing a program that constructs three-dimensional images using camera-based gesture inputs of a user, as recited in claim 11, wherein the three-dimensional graphical objects comprise a soccer ball having a set position (f), and a goal area having a set position on the video image display, the computer-readable memory device further comprising:

instructions for comparing a foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user;

instructions for moving the soccer ball according to the calculated foot velocity (fv);

instructions for slowing the soccer ball down by a predetermined velocity; and

instructions for sounding a bell if the soccer ball enters the goal area.

16. A computer program product for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system having a video image display connected to a microprocessor, the computer program product comprising:

means for receiving video signals indicative of gestures of the system user to determine the three-dimensional positions of the body and principle body parts of the system user;

means for constructing three-dimensional images of the system user and interaction area on the video image display based upon the received video signals; and

means for displaying, on the video image display, three-dimensional graphical objects superimposed to appear as if they occupied the interaction area, wherein movement by the system user causes apparent movement by the superimposed, three-dimensional objects displayed on the video image display.

17. A computer program product for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system, as recited in claim 16, wherein the superimposed, three-dimensional objects appear to move as if they were physical objects.

18. A computer program product for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system, as recited in claim 16, wherein the means for constructing the three-dimensional image comprises:

means for projecting two-dimensional positions (u, v) of the feet of the system user to three-dimensional coordinates (x, y, z) of the feet;

means for mapping the head and hands of the system user to three-dimensional coordinates assuming the head and hands are offset from a z position of the feet;

means for using a height (h) of the system user with biometric data to calculate the shoulder offset of the system user from the head, and to calculate the arm length of the system user;

means for calculating the offset of each arm of the system user from a corresponding foot of the system user; and

means for supplying the calculated three-dimensional positions of the head, hands and feet of the system user to the video image display.

19. A computer program product for constructing three-dimensional images using camera-based gesture inputs of a user of a computer system, as recited in claim 18, wherein the three-dimensional graphical objects comprise a soccer ball having a set position (f), and a goal area having a set position on the video image display, the computer program product further comprising:

means for comparing a foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user;

means for moving the soccer ball according to the calculated foot velocity (fv);

means for slowing the soccer ball down by a predetermined velocity; and

means for sounding a bell if the soccer ball enters the goal area.

20. A computer program product for constructing three-dimensional images using carmera-based gesture inputs of a user of a computer system, as recited in claim 16, wherein the three-dimensional graphical objects comprise a soccer ball having a set position (f), and a goal area having a set position on the video image display, the computer program product further comprising:

means for comparing a foot position (p) of the system user with the set position (f) of the soccer ball so to calculate the foot velocity (fv) of the system user;

means for moving the soccer ball according to the calculated foot velocity (fv);

means for slowing the soccer ball down by a predetermined velocity; and

means for sounding a bell if the soccer ball enters the goal area.