Video-based image control system

• US 8,274,535 B2
• Filed: 08/17/2007
• Issued: 09/25/2012
• Est. Priority Date: 07/24/2000
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan, MacDougall, Francis
• Primary Examiner(s): Wu, Xiao M.
• Assistant Examiner(s): Hoang, Phi
• Application Number:
  US11/840,550
• Publication Number:
  US 20080018595A1
• Time in Patent Office: 1,866 Days
• Field of Search: 463/37, 382/103
• US Class Current: 345/676
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of using stereo vision to interface with a computer is provided. The method includes capturing a stereo image, and processing the stereo image to determine position information of an object in the stereo image. The object is controlled by a user. The method also includes communicating the position information to the computer to allow the user to interact with a computer application.

Claims

1. A method comprising:

identifying features in an image of a scene;

determining a three-dimensional position for one or more features in the scene;

analyzing the three-dimensional position of at least one feature in the scene to determine position information of a control object;

causing display of a video display that includes at least a first virtual object and a second virtual object laid out at different depths, the first virtual object being defined by first three-dimensional bounds that describe a three-dimensional shape of the first virtual object and the second virtual object being defined by second three-dimensional bounds that describe a three-dimensional shape of the second virtual object;

mapping the position information of the control object to three-dimensional coordinates relative to the video display including a depth coordinate relative to the video display;

comparing the three-dimensional coordinates of the control object including the depth coordinate of the control object to the first three-dimensional bounds that define the first virtual object and the second three-dimensional bounds that define the second virtual object;

based on comparison results, determining whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object or the second three-dimensional bounds that define the second virtual object;

based on a determination that the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object, determining that the control object is interacting with the first virtual object and controlling an application based on the determination that control object is interacting with the first virtual object; and

based on a determination that the three-dimensional coordinates of the control object are within the second three-dimensional bounds that define the second virtual object, determining that the control object is interacting with the second virtual object and controlling the application based on the determination that control object is interacting with the second virtual object,

wherein determining whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object or the second three-dimensional bounds that define the second virtual object comprises;

determining that the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object when the depth coordinate of the control object indicates that the control object is located at a first depth relative to the video display, and

determining that the three-dimensional coordinates of the control object are within the second three-dimensional bounds that define the second virtual object when the depth coordinate of the control object indicates that the control object is located at a second depth relative to the video display, the second depth relative to the video display being different than the first depth relative to the video display.

2. The method of claim 1, wherein the position information is relative position information, relative to an object detection region.

3. The method of claim 1, further comprising:

defining an object detection region;

defining a sub-region within the object detection region,

wherein mapping the position information of the control object to the coordinates associated with the display further comprises;

linearly mapping the position information of the control object to coordinates within the display, if the position information is within the sub-region, and

mapping the position information to a border of the display, if the position information is outside of the sub-region and inside of the object detection region.

4. The method of claim 1, further comprising:

analyzing the three-dimensional position of features to determine position information of at least first and second objects; and

selecting the first object as the control object.

5. The method of claim 4, further comprising determining that the first object is closer to an image detector than the second object, wherein the first object is selected as the control object based on determining that the first object is closer to the image detector.

6. The method of claim 4, further comprising determining that the first object has largest or smallest world coordinate system coordinates, wherein the first object is selected as the control object based on determining that the first object has largest or smallest world coordinate system coordinates.

7. The method of claim 1, further comprising displaying an indicator on the display based on the coordinates.

8. The method of claim 7, further comprising displaying a control on the display, wherein the indicator is displayed over the control.

9. The method of claim 8, further comprising outputting a feedback signal indicating that the indicator is displayed over the control.

10. The method of claim 1, further comprising displaying an avatar representing the control object.

11. The method of claim 10, wherein the avatar comprises a virtual human-like form.

12. The method of claim 10, wherein the avatar is rendered in and interacts with a virtual environment forming part of the application.

13. The method of claim 12, further comprising comparing a position of a hand associated with the avatar to a position of a virtual object within the virtual environment, thereby enabling a user to interact with the virtual object.

14. The method of claim 10, wherein the position information of the control object is mapped to a virtual torso, a virtual hand, or a virtual head of the avatar.

15. The method of claim 10, further comprising moving the avatar to a neutral position based on analyzing the three-dimensional position information of the feature.

16. The method of claim 10, further comprising comparing the position information to a neutral position, wherein the coordinates are mapped based on a relative position between the position information and the neutral position.

17. The method of claim 16, wherein controlling the application further comprises causing the avatar to lean.

18. The method of claim 1,

wherein analyzing the three-dimensional position of the feature further comprises determining first and second position information of the control object, and

wherein the first position information is mapped to first coordinates and the second position information is mapped to second coordinates.

19. The method of claim 18, wherein the application is controlled based on a change between the first coordinates and the second coordinates.

20. The method of claim 19, wherein the application is controlled based on the change between the first coordinates and the second coordinates, over a predetermined duration.

21. The method of claim 19, further comprising:

mapping the change between the first coordinates and the second coordinates to a velocity function; and

controlling the application based on the velocity function.

22. The method of claim 19, wherein the change is indicative of a hovering control object maneuver.

23. The method of claim 19, wherein the change is indicative of a back-and-forth control object maneuver.

24. The method of claim 18, further comprising smoothing the first and second coordinates, thereby reducing jitter between consecutive image frames.

25. The method of claim 1, wherein analyzing the three-dimensional position of a feature determines a head position of a user.

26. The method of claim 1, further comprising calculating hand orientation information based on the coordinates.

27. The method of claim 26, wherein the application is controlled based on the hand orientation information.

28. The method of claim 26, further including smoothing changes in the hand orientation information.

29. The method of claim 1 further comprising defining an object detection region, wherein the object detection region is a dynamic object detection region.

30. The method of claim 29 wherein defining the dynamic object detection region further comprises:

establishing the position of a top of a head of a user based upon a topmost feature of the scene description, unless the topmost feature is at a boundary of the scene; and

determining the position of a hand detection region relative to the top of the head of the user.

31. The method of claim 1, further comprising:

identifying a position of a torso-divisioning plane based on features in the scene; and

determining a position of a hand detection region relative to the torso-divisioning plane on an axis perpendicular to the torso divisioning plane.

32. The method of claim 31, further comprising:

identifying a body center position and a body boundary position based on features in the scene;

identifying a position indicating part of an arm of a user based on features in the scene, using the intersection of the feature with the torso divisioning plane; and

identifying the arm as either a left arm or a right arm based on the position indicating part of the arm relative to the body center position.

33. The method of claim 32 further comprising defining an object detection region, wherein the object detection region is a dynamic object detection region, and wherein a position of the dynamic object detection region is defined based on the body center position.

34. The method of claim 33, wherein defining the dynamic object detection region further comprises establishing a shoulder position from the body center position, the body boundary position, the torso-divisioning plane, and the left arm or the right arm identification.

35. The method of claim 34, wherein defining the dynamic object detection region further includes determining position data for the hand detection region relative to the shoulder position.

36. The method of claim 34 further including:

determining the position of the dynamic object detection region relative to the torso divisioning plane in the axis perpendicular to the torso divisioning plane;

determining the position of the dynamic object detection region in the horizontal axis relative to the shoulder position; and

determining the position of the dynamic object detection region in the vertical axis relative to an overall height of the user using the body boundary position.

37. The method of claim 35, further comprising smoothing the position data for the hand detection region.

38. The method of claim 1, wherein the first virtual object is defined by first spherical bounds that describe a spherical shape of the first virtual object and the second virtual object is defined by second spherical bounds that describe a spherical shape of the second virtual object.

39. The method of claim 1, wherein the first virtual object is defined by first cubic bounds that describe a cubic shape of the first virtual object and the second virtual object is defined by second cubic bounds that describe a cubic shape of the second virtual object.

40. The method of claim 1:

wherein causing display of a video display that includes at least a first virtual object and a second virtual object laid out at different depths comprises causing display of a video display that includes at least a first virtual folder and a second virtual folder laid out at different depths, the first virtual folder being defined by first three-dimensional bounds that describe a three-dimensional shape of the first virtual folder and the second virtual folder being defined by second three-dimensional bounds that describe a three-dimensional shape of the second virtual folder;

wherein comparing the three-dimensional coordinates of the control object to the first three-dimensional bounds that define the first virtual object and the second three-dimensional bounds that define the second virtual object comprises comparing the three-dimensional coordinates of the control object to the first three-dimensional bounds that define the first virtual folder and the second three-dimensional bounds that define the second virtual folder;

wherein determining whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object or the second three-dimensional bounds that define the second virtual object comprises determining whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual folder or the second three-dimensional bounds that define the second virtual folder;

wherein determining that the control object is interacting with the first virtual object and controlling an application based on the determination that control object is interacting with the first virtual object comprises determining that the control object is interacting with the first virtual folder and controlling an application based on the determination that control object is interacting with the first virtual folder; and

wherein determining that the control object is interacting with the second virtual object and controlling the application based on the determination that control object is interacting with the second virtual object comprises determining that the control object is interacting with the second virtual folder and controlling the application based on the determination that control object is interacting with the second virtual folder.

41. A device comprising:

an image detector configured to receive an image of a scene; and

a processor configured to;

identify features in the image,

determine a three-dimensional position for one or more features in the scene,

analyze the three-dimensional position of at least one feature in the scene to determine position information of a control object,

cause display of a video display that includes at least a first virtual object and a second virtual object laid out at different depths, the first virtual object being defined by first three-dimensional bounds that describe a three-dimensional shape of the first virtual object and the second virtual object being defined by second three-dimensional bounds that describe a three-dimensional shape of the second virtual object;

map the position information of the control object to three-dimensional coordinates relative to the video display including a depth coordinate relative to the video display;

compare the three-dimensional coordinates of the control object including the depth coordinate of the control object to the first three-dimensional bounds that define the first virtual object and the second three-dimensional bounds that define the second virtual object;

based on comparison results, determine whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object or the second three-dimensional bounds that define the second virtual object;

based on a determination that the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object, determine that the control object is interacting with the first virtual object and control an application based on the determination that control object is interacting with the first virtual object; and

based on a determination that the three-dimensional coordinates of the control object are within the second three-dimensional bounds that define the second virtual object, determine that the control object is interacting with the second virtual object and

control the application based on the determination that control object is interacting with the second virtual object,

wherein the processor is configured to determine whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object or the second three-dimensional bounds that define the second virtual object by;

determining that the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object when the depth coordinate of the control object indicates that the control object is located at a first depth relative to the video display, and

determining that the three-dimensional coordinates of the control object are within the second three-dimensional bounds that define the second virtual object when the depth coordinate of the control object indicates that the control object is located at a second depth relative to the video display, the second depth relative to the video display being different than the first depth relative to the video display.

42. A system comprising:

means for identifying features in an image of a scene;

means for determining a three-dimensional position for one or more features in the scene;

means for analyzing the three-dimensional position of at least one feature in the scene to determine position information of a control object;

means for causing display of a video display that includes at least a first virtual object and a second virtual object laid out at different depths, the first virtual object being defined by first three-dimensional bounds that describe a three-dimensional shape of the first virtual object and the second virtual object being defined by second three-dimensional bounds that describe a three-dimensional shape of the second virtual object;

means for mapping the position information of the control object to three-dimensional coordinates relative to the video display including a depth coordinate relative to the video display;

means for comparing the three-dimensional coordinates of the control object including the depth coordinate of the control object to the first three-dimensional bounds that define the first virtual object and the second three-dimensional bounds that define the second virtual object;

means for determining, based on comparison results, whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object or the second three-dimensional bounds that define the second virtual object;

means for determining, based on a determination that the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object, that the control object is interacting with the first virtual object and controlling an application based on the determination that control object is interacting with the first virtual object; and

means for determining, based on a determination that the three-dimensional coordinates of the control object are within the second three-dimensional bounds that define the second virtual object, that the control object is interacting with the second virtual object and controlling the application based on the determination that control object is interacting with the second virtual object,

wherein the means for determining whether the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object or the second three-dimensional bounds that define the second virtual object further comprises;

means for determining that the three-dimensional coordinates of the control object are within the first three-dimensional bounds that define the first virtual object when the depth coordinate of the control object indicates that the control object is located at a first depth relative to the video display, and

means for determining that the three-dimensional coordinates of the control object are within the second three-dimensional bounds that define the second virtual object when the depth coordinate of the control object indicates that the control object is located at a second depth relative to the video display, the second depth relative to the video display being different than the first depth relative to the video display.

43. The system of claim 42, wherein the position information is relative position information, relative to an object detection region.

44. The system of claim 42, further comprising:

means for defining an object detection region; and

means for defining a sub-region within the object detection region,

wherein the means for mapping the position information of the control object to the coordinates associated with the display further comprises;

means for linearly mapping the position information of the control object to coordinates within the display, if the position information is within the sub-region, and

means for mapping the position information to a border of the display, if the position information is outside of the sub-region and inside of the object detection region.

45. The system of claim 42, further comprising:

means for analyzing the three-dimensional position of features to determine position information of at least first and second objects; and

means for selecting the first object as the control object.

46. The system of claim 42, further comprising means for displaying an indicator on the display based on the coordinates.

47. The system of claim 42, further comprising means for displaying an avatar representing the control object.

48. The system of claim 42,

wherein the means for analyzing the three-dimensional position of the at least one feature further comprises means for determining first and second position information of the control object, and

wherein the first position information is mapped to first coordinates and the second position information is mapped to second coordinates.