VIDEO-BASED IMAGE CONTROL SYSTEM

• US 20080018595A1
• Filed: 08/17/2007
• Published: 01/24/2008
• Est. Priority Date: 07/24/2000
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: MacDougall, Francis, Hildreth, Evan
• Granted Patent: US 8,274,535 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of using stereo vision to interface with a computer is provided. The method includes capturing a stereo image, and processing the stereo image to determine position information of an object in the stereo image. The object is controlled by a user. The method also includes communicating the position information to the computer to allow the user to interact with a computer application.

Claims

1. A method comprising:

filtering an image of a scene into features;

generating a scene description indicating a three-dimensional position for each feature in the scene;

defining an object detection region;

analyzing the three-dimensional position of a feature within the object detection region to determine first and second position information of a control object at first and second times, respectively;

mapping a change in the first and second position information of the control object to a virtual torso, a virtual hand, or a virtual head of an avatar;

displaying the avatar on a display, the avatar being rendered in and interacting with a virtual environment forming part of a computer application and comprising a virtual human-like form;

comparing a position of the avatar to a position of a virtual object within the virtual environment, thereby enabling an interaction with the virtual object; and

controlling the computer application based on the interaction.

2. A method comprising:

filtering an image of a scene into features;

generating a scene description indicating a three-dimensional position for each feature in the scene;

defining an object detection region;

analyzing the three-dimensional position of a feature within the object detection region to determine position information of a control object;

mapping the position information of the control object to coordinates associated with a display; and

controlling a computer application based on the coordinates.

3. The method of claim 2, wherein the position information is relative position information, relative to the object detection region.

4. The method of claim 2, further comprising:

defining a sub-region within the object detection region,

wherein mapping the position information of the control object to the coordinates associated with the display further comprises;

linearly mapping the position information of the control object to coordinates within the display, if the position information is within the sub-region, and

mapping the position information to a border of the display, if the position information is outside of the sub-region and inside of the object detection region.

5. The method of claim 2, further comprising:

analyzing the three-dimensional position of the features within the object detection region to determine position information of at least first and second objects; and

selecting the first object as the control object.

6. The method of claim 5, further comprising determining that the first object is closer to an image detector than the second object, wherein the first object is selected as the control object based on determining that the first object is closer to the image detector.

7. The method of claim 5, further comprising determining that the first object has largest or smallest world coordinate system coordinates, wherein the first object is selected as the control object based on determining that the first object has largest or smallest world coordinate system coordinates.

8. The method of claim 2, further comprising displaying an indicator on the display based on the coordinates.

9. The method of claim 8, further comprising displaying a control on the display, wherein the indicator is displayed over the control.

10. The method of claim 9, further comprising outputting a feedback signal indicating that the indicator is displayed over the control.

11. The method of claim 9, wherein displaying the indicator further comprises:

comparing the coordinates to bounds associated with a control; and

displaying the indicator over the control if the coordinates are within the bounds associated with the control.

12. The method of claim 11, wherein the bounds are spherical or cubic bounds.

13. The method of claim 2, further comprising displaying an avatar representing the control object.

14. The method of claim 13, wherein the avatar comprises a virtual human-like form.

15. The method of claim 13, wherein the avatar is rendered in and interacts with a virtual environment forming part of the computer application.

16. The method of claim 15, further comprising comparing a position of a hand associated with the avatar to a position of a virtual object within the virtual environment, thereby enabling a user to interact with the virtual object.

17. The method of claim 13, wherein the position information of the control object is mapped to a virtual torso, a virtual hand, or a virtual head of the avatar.

18. The method of claim 13, further comprising moving the avatar to a neutral position based on analyzing the three-dimensional position information of the feature.

19. The method of claim 13, further comprising comparing the position information to a neutral position, wherein the coordinates are mapped based on a relative position between the position information and the neutral position.

20. The method of claim 19, wherein controlling the computer application further comprises causing the avatar to lean.

21. The method of claim 2,

wherein analyzing the three-dimensional position of the feature further comprises determining first and second position information of the control object, and

wherein the first position information is mapped to first coordinates and the second position information is mapped to second coordinates.

22. The method of claim 21, wherein the computer application is controlled based on a change between the first coordinates and the second coordinates.

23. The method of claim 22, wherein the computer application is controlled based on the change between the first coordinates and the second coordinates, over a predetermined duration.

24. The method of claim 22, further comprising:

mapping the change between the first coordinates and the second coordinates to a velocity function; and

controlling the computer application based on the velocity function.

25. The method of claim 22, wherein the change is indicative of a hovering control object maneuver.

26. The method of claim 22, wherein the change is indicative of a back-and-forth control object maneuver.

27. The method of claim 21, further comprising smoothing the first and second coordinates, thereby reducing jitter between consecutive image frames.

28. The method of claim 2, wherein analyzing the three-dimensional position of a feature determines a head position of a user.

29. The method of claim 2, further comprising calculating hand orientation information based on the coordinates.

30. The method of claim 29, wherein the computer application is controlled based on the hand orientation information.

31. The method of claim 29, further including smoothing changes in the hand orientation information.

32. The method of claim 2, wherein the object detection region is a dynamic object detection region.

33. The method of claim 2, further comprising:

identifying a position of a torso-divisioning plane based on the scene description; and

determining a position of a hand detection region relative to the torso-divisioning plane on an axis perpendicular to the torso divisioning plane.

34. The method of claim 33, further comprising:

identifying a body center position and a body boundary position based on the scene description;

identifying a position indicating part of an arm of a user based on the scene description, using the intersection of the feature with the torso divisioning plane; and

identifying the arm as either a left arm or a right arm based on the position indicating part of the arm relative to the body center position.

35. The method of claim 34, wherein the object detection region is a dynamic object detection region, and wherein a position of the dynamic object detection region is defined based on the body center position.

36. The method of claim 35, wherein defining the dynamic object detection region further comprises establishing a shoulder position from the body center position, the body boundary position, the torso-divisioning plane, and the left arm or the right arm identification.

37. The method of claim 36, wherein defining the dynamic object detection region further includes determining position data for the hand detection region relative to the shoulder position.

38. The method of claim 37, further comprising smoothing the position data for the hand detection region.

39. The method of claim 36 further including:

determining the position of the dynamic object detection region relative to the torso divisioning plane in the axis perpendicular to the torso divisioning plane;

determining the position of the dynamic object detection region in the horizontal axis relative to the shoulder position; and

determining the position of the dynamic object detection region in the vertical axis relative to an overall height of the user using the body boundary position.

40. The method of claim 3 wherein defining the dynamic object detection region further comprising:

establishing the position of a top of a head of a user based upon a topmost feature of the scene description, unless the topmost feature is at a boundary of the scene description; and

determining the position of a hand detection region relative to the top of the head of the user.

41. A device comprising:

an image detector configured to receive an image of a scene; and

a processor configured to;

filter the image into features,

generate a scene description indicating a three-dimensional position for each feature in the scene,

define an object detection region,

analyze the three-dimensional position of a feature within the object detection region to determine position information of a control object,

map the position information of the control object to coordinates associated with a display, and

control a computer application based on the coordinates.

42. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause data processing apparatus to:

filter an image of a scene into features;

generate a scene description indicating a three-dimensional position for each feature in the scene;

define an object detection region;

analyze the three-dimensional position of a feature within the object detection region to determine position information of a control object;

map the position information of the control object to coordinates associated with a display; and

control a computer application based on the coordinates.