Video-based image control system

• US 8,624,932 B2
• Filed: 08/23/2012
• Issued: 01/07/2014
• Est. Priority Date: 07/24/2000
• Status: Expired due to Term

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan, MacDougall, Francis
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US13/593,355
• Publication Number:
  US 20130147790A1
• Time in Patent Office: 502 Days
• Field of Search: 345/158, 382/103
• US Class Current: 345/676
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of using stereo vision to interface with a computer is provided. The method includes capturing a stereo image, and processing the stereo image to determine position information of an object in the stereo image. The object is controlled by a user. The method also includes communicating the position information to the computer to allow the user to interact with a computer application.

Claims

1. A method comprising:

determining position information of a control object in an image;

causing a display to display a first virtual object and a second virtual object located at a different depth than the first virtual object;

mapping the position information of the control object to three-dimensional coordinates, wherein the three-dimensional coordinates comprise a depth coordinate;

determining that the three-dimensional coordinates of the control object are within three-dimensional bounds of the first virtual object if the depth coordinate of the control object indicates that the control object is located at a first depth relative to the display;

determining that the three-dimensional coordinates of the control object are within three-dimensional bounds of the second virtual object if the depth coordinate of the control object indicates that the control object is located at a second depth relative to the display;

creating a history of three-dimensional coordinates using position information of the control object, wherein;

the history is for a predefined duration of time, and

the history comprises three-dimensional coordinates associated with the control object from during the duration of time;

identifying a minimum position and maximum position in each dimension of the history;

determining a distance between the minimum position and the maximum position for each dimension of the history;

for each dimension, determining that the distance is within a predefined threshold distance; and

after determining that the distance is within the predefined threshold distance for each dimension, outputting an indication of a hover gesture.

2. The method of claim 1, further comprising:

controlling an application based on the three-dimensional coordinates of the control object being determined to be within the three-dimensional bounds of the first virtual object.

3. The method of claim 1, further comprising:

controlling an application based on the three-dimensional coordinates of the control object being determined to be within the three-dimensional bounds of the second virtual object.

4. The method of claim 1, further comprising:

displaying, by the display, an avatar representing the control object.

5. The method of claim 4, wherein the avatar comprises a virtual human-like form.

6. The method of claim 1, wherein determining the position information of the control object in the image comprises:

identifying features in the image, wherein the image is of a scene;

determining a three-dimensional position for each feature of a plurality of features in the scene; and

analyzing the three-dimensional position of each feature of the plurality of features to determine position information of the control object.

7. The method of claim 6, further comprising:

identifying a position of a torso-divisioning plane based on the features in the scene; and

determining a position of a hand detection region relative to the torso-divisioning plane.

8. The method of claim 7, further comprising:

identifying a body center position and a body boundary position based on the features in the scene;

identifying an arm position indicating at least part of an arm based on the features in the scene; and

identifying the arm as either a left arm or a right arm based on the arm position indicating at least the part of the arm relative to the body center position.

9. The method of claim 1, further comprising:

calculating hand orientation information based on the three-dimensional coordinates of the control object.

10. The method of claim 1, further comprising:

selecting an object from a plurality of objects present in the image as the control object, wherein selection of the object is at least partially based on the object being within an object detection region and being closest to one or more cameras used to capture the image.

11. The method of claim 1, further comprising:

mapping the three-dimensional coordinates of the control object relative to a neutral control point position; and

transforming the mapped three-dimensional coordinates to a velocity function.

12. The method of claim 11, further comprising:

using the velocity function to move a viewpoint associated with the display.

13. A system comprising:

one or more processors; and

a memory communicatively coupled with and readable by the one or more processors and having stored therein processor-readable instructions which, when executed by the one or more processors, cause the one or more processors to;

determine position information of a control object in an image;

cause a display to display a first virtual object and a second virtual object located at a different depth than the first virtual object;

map the position information of the control object to three-dimensional coordinates, wherein the three-dimensional coordinates comprise a depth coordinate;

determine that the three-dimensional coordinates of the control object are within three-dimensional bounds of the first virtual object if the depth coordinate of the control object indicates that the control object is located at a first depth relative to the display;

determine that the three-dimensional coordinates of the control object are within three-dimensional bounds of the second virtual object if the depth coordinate of the control object indicates that the control object is located at a second depth relative to the display;

create a history of three-dimensional coordinates using position information of the control object, wherein;

the history is for a predefined duration of time, and

the history comprises three-dimensional coordinates associated with the control object from during the duration of time;

identify a minimum position and maximum position in each dimension of the history;

determine a distance between the minimum position and the maximum position for each dimension of the history;

for each dimension, determine that the distance is within a predefined threshold distance; and

after determining that the distance is within the predefined threshold distance for each dimension, output an indication of a hover gesture.

14. The system of claim 13, wherein the processor-readable instructions, when executed by the one or more processors, further cause the one or more processors to:

control an application based on the three-dimensional coordinates of the control object being determined to be within the three-dimensional bounds of the first virtual object.

15. The system of claim 13, wherein the processor-readable instructions, which when executed by the one or more processors, further cause the one or more processors to:

control an application based on the three-dimensional coordinates of the control object being determined to be within the three-dimensional bounds of the second virtual object.

16. The system of claim 13, wherein the processor-readable instructions, which when executed by the one or more processors, further cause the one or more processors to:

cause an avatar representing the control object to be displayed.

17. The system of claim 16, wherein the avatar comprises a virtual human-like form.

18. The system of claim 13, wherein the processor-readable instructions that cause the one or more processors to determine the position information of the control object in the image further comprise processor-readable instructions, which when executed by the one or more processors, cause the one or more processors to:

identify features in the image, wherein the image is of a scene;

determine a three-dimensional position for each feature of a plurality of features in the scene; and

analyze the three-dimensional position of each feature of the plurality of features to determine position information of the control object.

19. The system of claim 18, wherein the processor-readable instructions, when executed by the one or more processors, further cause the one or more processors to:

identify a position of a torso-divisioning plane based on the features in the scene; and

determine a position of a hand detection region relative to the torso-divisioning plane.

20. The system of claim 19, wherein the processor-readable instructions, when executed by the one or more processors, further cause the one or more processors to:

identify a body center position and a body boundary position based on the features in the scene;

identify an arm position indicating at least part of an arm based on the features in the scene; and

identify the arm as either a left arm or a right arm based on the arm position indicating at least the part of the arm relative to the body center position.

21. The system of claim 13, wherein the processor-readable instructions, when executed by the one or more processors, further cause the one or more processors to:

calculate hand orientation information based on the three-dimensional coordinates of the control object.

22. The system of claim 13, wherein the processor-readable instructions, when executed by the one or more processors, further cause the one or more processors to:

select an object from a plurality of objects present in the image as the control object, wherein selection of the object is at least partially based on the object being within an object detection region and being closest to one or more cameras used to capture the image.

23. The system of claim 13, wherein the processor-readable instructions, when executed by the one or more processors, further cause the one or more processors to:

map the three-dimensional coordinates of the control object relative to a neutral control point position; and

transform the mapped three-dimensional coordinates to a velocity function.

24. The system of claim 23, wherein the processor-readable instructions, when executed by the one or more processors, further cause the one or more processors to:

use the velocity function to move a viewpoint associated with the display.

25. A non-transitory processor-readable storage medium, the medium comprising processor-readable instructions configured to cause a processing system to:

determine position information of a control object in an image;

cause a display to display a first virtual object and a second virtual object located at a different depth than the first virtual object;

map the position information of the control object to three-dimensional coordinates, wherein the three-dimensional coordinates comprise a depth coordinate;

determine that the three-dimensional coordinates of the control object are within three-dimensional bounds of the first virtual object if the depth coordinate of the control object indicates that the control object is located at a first depth relative to the display;

determine that the three-dimensional coordinates of the control object are within three-dimensional bounds of the second virtual object if the depth coordinate of the control object indicates that the control object is located at a second depth relative to the display;

create a history of three-dimensional coordinates using position information of the control object, wherein;

the history is for a predefined duration of time, and

the history comprises three-dimensional coordinates associated with the control object from during the duration of time;

identify a minimum position and maximum position in each dimension of the history;

determine a distance between the minimum position and the maximum position for each dimension of the history;

for each dimension, determine that the distance is within a predefined threshold distance; and

after determining that the distance is within the predefined threshold distance for each dimension, output an indication of a hover gesture.

26. The medium of claim 25, wherein the processor-readable instructions are further configured to cause a processing system to:

control an application based on the three-dimensional coordinates of the control object being determined to be within the three-dimensional bounds of the first virtual object.

27. An apparatus comprising:

means for determining position information of a control object in an image;

means for causing a display to display a first virtual object and a second virtual object located at a different depth than the first virtual object;

means for mapping the position information of the control object to three-dimensional coordinates, wherein the three-dimensional coordinates comprise a depth coordinate;

means for determining that the three-dimensional coordinates of the control object are within three-dimensional bounds of the first virtual object if the depth coordinate of the control object indicates that the control object is located at a first depth relative to the display;

means for determining that the three-dimensional coordinates of the control object are within three-dimensional bounds of the second virtual object if the depth coordinate of the control object indicates that the control object is located at a second depth relative to the display;

means for creating a history of three-dimensional coordinates using position information of the control object, wherein;

the history is for a predefined duration of time, and

the history comprises three-dimensional coordinates associated with the control object from during the duration of time;

means for identifying a minimum position and maximum position in each dimension of the history;

means for determining a distance between the minimum position and the maximum position for each dimension of the history;

means for determining that the distance is within a predefined threshold distance for each dimension; and

means for outputting an indication of a hover gesture after determining that the distance is within the predefined threshold distance for each dimension.

28. The apparatus of claim 27, further comprising:

means for controlling an application based on the three-dimensional coordinates of the control object being determined to be within the three-dimensional bounds of the first virtual object.