Video-based image control system

• US 7,898,522 B2
• Filed: 06/01/2007
• Issued: 03/01/2011
• Est. Priority Date: 07/24/2000
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: MacDougall, Francis, Hildreth, Evan
• Primary Examiner(s): Dharia; Prabodh M
• Application Number:
  US11/757,180
• Publication Number:
  US 20080030460A1
• Time in Patent Office: 1,369 Days
• Field of Search: 345/2.1, 345156-158, 345/161, 345/173, 345/356, 345/419, 345/421, 345/473, 345/757, 382/103, 382/154, 382254-260, 382/173, 348/143, 348/169, 348/219.1, 348/48, 340/901, 701/1, 703/2
• US Class Current: 345/156
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of using stereo vision to interface with computer is provided. The method includes capturing a stereo image, and processing the stereo image to determine position information of an object in the stereo image. The object is controlled by a user. The method also includes communicating the position information to the computer to allow the user to interact with a computer application.

Claims

1. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

receive three dimensional data points describing depth characteristics of a user;

define a plane that passes through a torso of the user'"'"'s body based on the received three dimensional data points;

determine a boundary of a hand detection region that has an orientation parallel to the plane and that is defined to be a predetermined distance in front of the plane along an axis that is defined normal to the plane;

detect a position of a hand of the user'"'"'s body within the hand detection region; and

control an application based on the detected position of the hand.

2. The computer program product of claim 1, further comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

determine, based on defining the plane, a position of the torso;

determine an expected range of motion of the hand of the user'"'"'s body; and

define the hand detection region based on the expected range of motion.

3. The computer program product of claim 1, further comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

determine, based on defining the plane, a position of the torso; and

identify, based on the determined positions of the torso and the hand, whether the hand is a left hand or a right hand,

wherein controlling the application further comprises generating first or second user inputs to the application if the hand is identified as a right hand or a left hand, respectively.

4. The computer program product of claim 1, wherein:

defining the plane further comprises defining a plane segmenting the torso from left and right hands of the user'"'"'s body;

determining the position of the hand further comprises determining positions of the left and right hands; and

controlling an application further comprises causing an avatar, including left and right indicators corresponding to the left and right hands, respectively, to move within a representation of a three-dimensional virtual environment based on the determined positions of the left and right hands.

5. The computer program product of claim 1, wherein:

determining the position of the hand further comprises determining a change in position of the hand; and

controlling the application further comprises mapping the change in position to a user input.

6. The computer program product of claim 1, further comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

defining a second plane parallel to the plane, on an opposite side of the plane as the torso, wherein controlling the application further comprises;

generating a first user input if the position of the hand is determined to be between the plane and the second plane, and

generating a second, different user input of the position of the hand is determined to be on an opposite side of the plane as the second plane.

7. The computer program product of claim 1, further comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

solve a position of the hand detection region on each of three axes.

8. The computer program product of claim 1, further comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

find a position of a top of the user'"'"'s head;

position the hand detection region a predefined distance below the top of the user'"'"'s head.

9. The computer program product of claim 1, further comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

bias a position of the hand detection region toward an arm of the user'"'"'s body.

10. The computer program product of claim 1, wherein determining the position of the hand of the user'"'"'s body within the hand detection region comprises determining the position of the hand as a position of a feature within the hand detection region that is furthest from the boundary of the hand detection region.

11. The computer program product of claim 1, wherein determining the boundary of the hand detection region comprises determining a position of the hand detection region along a z-axis as a predetermined distance in front of the user'"'"'s torso, further comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

determine a position of the hand detection region along a y-axis as a predefined distance below a top of the user'"'"'s head; and

determine a position of the hand detection region along an x-axis based on an expected range of motion of the user'"'"'s hand.

12. A device comprising:

a storage medium configured to store three-dimensional coordinates that express positions of a user'"'"'s body relative to an image detector; and

a processor configured to;

define a plane that passes through a torso of the user'"'"'s body;

determine a boundary of a hand detection region that has an orientation parallel to the plane and that is defined to be a predetermined distance in front of the plane along an axis that is defined normal to the plane;

determine a position of a hand of the user'"'"'s body within the hand detection region; and

control an application based on the determined position of the hand.

13. The device of claim 12, wherein the processor is further configured to:

determine, based on defining the plane, a position of the torso;

determine an expected range of motion of the hand of the user'"'"'s body; and

define the hand detection region based on the expected range of motion.

14. The device of claim 12, wherein the processor is further configured to:

determine, based on defining the plane, a position of the torso; and

identify, based on the determined positions of the torso and the hand, whether the hand is a left hand or a right hand,

wherein controlling the application further comprises generating first or second user inputs to the application if the hand is identified as a right hand or a left hand, respectively.

15. The device of claim 12, wherein:

determining the position of the hand further comprises determining a change in position of the hand; and

controlling the application further comprises mapping the change in position to a user input.

16. The device of claim 12, wherein the processor is further configured to:

solve a position of the hand detection region on each of three axes.

17. The device of claim 12, wherein the processor is further configured to:

find a position of a top of the user'"'"'s head;

position the hand detection region a predefined distance below the top of the user'"'"'s head.

18. The device of claim 12, wherein the processor is further configured to:

bias a position of the hand detection region toward an arm of the user'"'"'s body.

19. The device of claim 12, wherein determining the position of the hand of the user'"'"'s body within the hand detection region comprises determining the position of the hand as a position of a feature within the hand detection region that is furthest from the boundary of the hand detection region.

20. The device of claim 12, wherein determining the boundary of the hand detection region comprises determining a position of the hand detection region along a z-axis as a predetermined distance in front of the user'"'"'s torso, wherein the processor is further configured to:

determine a position of the hand detection region along a y-axis as a predefined distance below a top of the user'"'"'s head; and

determine a position of the hand detection region along an x-axis based on an expected range of motion of the user'"'"'s hand.

21. A method comprising:

receiving three dimensional data points describing depth characteristics of a user;

defining a plane that passes through a torso of the user'"'"'s body based on the received three dimensional data points;

determining a boundary of a hand detection region that has an orientation parallel to the plane and that is defined to be a predetermined distance in front of the plane along an axis that is defined normal to the plane;

detecting a position of a hand of the user'"'"'s body within the hand detection region; and

controlling, by at least one processor, an application based on the detected position of the hand.

22. The method of claim 21, further comprising:

determining, based on defining the plane, a position of the torso;

determining an expected range of motion of the hand of the user'"'"'s body; and

defining the hand detection region based on the expected range of motion.

23. The method of claim 21, further comprising:

determining, based on defining the plane, a position of the torso; and

identifying, based on the determined positions of the torso and the hand, whether the hand is a left hand or a right hand,

wherein controlling the application further comprises generating first or second user inputs to the application if the hand is identified as a right hand or a left hand, respectively.

24. The method of claim 21, wherein:

defining the plane further comprises defining a plane segmenting the torso from left and right hands of the user'"'"'s body;

determining the position of the hand further comprises determining positions of the left and right hands; and

controlling an application further comprises causing an avatar, including left and right indicators corresponding to the left and right hands, respectively, to move within a representation of a three-dimensional virtual environment based on the determined positions of the left and right hands.

25. The method of claim 21, wherein:

determining the position of the hand further comprises determining a change in position of the hand; and

controlling the application further comprises mapping the change in position to a user input.

26. The method of claim 21, further comprising:

defining a second plane parallel to the plane, on an opposite side of the plane as the torso, wherein controlling the application further comprises;

generating a first user input if the position of the hand is determined to be between the plane and the second plane, and

generating a second, different user input of the position of the hand is determined to be on an opposite side of the plane as the second plane.

27. The method of claim 21, further comprising:

solving a position of the hand detection region on each of three axes.

28. The method of claim 21, further comprising:

finding a position of a top of the user'"'"'s head;

positioning the hand detection region a predefined distance below the top of the user'"'"'s head.

29. The method of claim 21, further comprising:

biasing a position of the hand detection region toward an arm of the user'"'"'s body.

30. The method of claim 21, wherein determining the position of the hand of the user'"'"'s body within the hand detection region comprises determining the position of the hand as a position of a feature within the hand detection region that is furthest from the boundary of the hand detection region.

31. The method of claim 21, wherein determining the boundary of the hand detection region comprises determining a position of the hand detection region along a z-axis as a predetermined distance in front of the user'"'"'s torso, further comprising:

determining a position of the hand detection region along a y-axis as a predefined distance below a top of the user'"'"'s head; and

determining a position of the hand detection region along an x-axis based on an expected range of motion of the user'"'"'s hand.