Video hand image-three-dimensional computer interface with multiple degrees of freedom

• US 6,147,678 A
• Filed: 12/09/1998
• Issued: 11/14/2000
• Est. Priority Date: 12/09/1998
• Status: Expired due to Term

• Current Assignee: WSOU Investments, LLC (WSOU Holdings, LLC)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Segen, Jakub, Kumar, Sentihil
• Primary Examiner(s): Saras, Steven J.
• Assistant Examiner(s): KUMAR, SRILAKSHMI K
• Application Number:
  US09/208,196
• Time in Patent Office: 706 Days
• Field of Search: 382/48, 382/168, 382/288, 395/155, 345/156, 345/168, 345/158, 364/188
• US Class Current: 345/158
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A video gesture-based three-dimensional computer interface system that uses images of hand gestures to control a computer and that tracks motion of the user'"'"'s hand or a portion thereof in a three-dimensional coordinate system with ten degrees of freedom. The system includes a computer with image processing capabilities and at least two cameras connected to the computer. During operation of the system, hand images from the cameras are continually converted to a digital format and input to the computer for processing. The results of the processing and attempted recognition of each image are then sent to an application or the like executed by the computer for performing various functions or operations. When the computer recognizes a hand gesture as a "point" gesture with one or two extended fingers, the computer uses information derived from the images to track three-dimensional coordinates of each extended finger of the user'"'"'s hand with five degrees of freedom. The computer utilizes two-dimensional images obtained by each camera to derive three-dimensional position (in an x, y, z coordinate system) and orientation (azimuth and elevation angles) coordinates of each extended finger.

Claims

1. A method of tracking, in a real world scene, three-dimensional position coordinates and orientation angles of a user'"'"'s hand and of the user'"'"'s extended index finger having a first axis and of an extended thumb having a second axis said method being implemented in a computer system having a first video acquisition device and a second video acquisition device for monitoring, from different positions, an identification zone defined in the real world scene, said method comprising the steps of:

(a) acquiring a first image of the user'"'"'s hand from the first video acquisition device and simultaneously acquiring a second image of the user'"'"'s hand from the second video acquisition device when the user'"'"'s hand is disposed within the identification zone;

(b) analyzing said first and second images to determine whether both said first and second images correspond to a pointing gesture of the user'"'"'s hand wherein at least the index finger is extended, and

(1) when both said first and second images are determined to correspond to the pointing gesture of the user'"'"'s hand;

(A) identifying a first end portion of the extended index finger on each of said first and second images; and

(B) determining whether the thumb is extended, and when the thumb is determined to be extended, identifying a second end portion of the extended thumb on each of said first and second images;

(2) when only one of said first and second images is determined to correspond to the pointing gesture of the user'"'"'s hand, repeating said step (a) and then said step (b); and

(3) when both said first and second images are determined to not correspond to the pointing gesture of the user'"'"'s hand, repeating said step (a) and then said step (b);

(c) determining a first set of pixel coordinates and a first orientation angle for the first end portion of the extended index finger from said first image and determining a second set of pixel coordinates and a second orientation angle for the first end portion of the extended index finger from said second image;

(d) defining a first virtual line in the real world scene in accordance with said first set of pixel coordinates and said first orientation angle and corresponding to the first axis of the extended index finger as shown in said first image, and defining a second virtual line in the real world scene in accordance with said second set of pixel coordinates and said second orientation angle and corresponding to the second axis of the extended index finger as shown in said second image;

(e) determining the three-dimensional position coordinates of the extended index finger first end portion by identifying coordinates of a first virtual intersection of said first and second virtual lines;

(f) determining a first parameter vector representative of a first linear projection of the index finger first axis in said first image in accordance with said first set of pixel coordinates and said first orientation angle, and determining a second parameter vector representative of a second linear projection of the index finger first axis in said second image in accordance with said second set of pixel coordinates and said second orientation angle;

(g) defining a first virtual plane along said first parameter vector, and defining a second virtual plane along said second parameter vector;

(h) determining the three-dimensional orientation angles of the extended index finger by identifying orientation parameters of a first intersecting line defined by an intersection of said first and said second virtual planes;

(i) when the thumb is determined to be extended at said step (A), determining a third set of pixel coordinates and a third orientation angle for the second end portion of the extended thumb from said first image and determining a fourth set of pixel coordinates and a fourth orientation angle for the second end portion of the extended thumb from said second image;

(j) defining a third virtual line in the real world scene in accordance with said third set of pixel coordinates and said third orientation angle and corresponding to the second axis of the extended thumb as shown in said second image, and defining a fourth virtual line in the real world scene in accordance with said fourth set of pixel coordinates and said fourth orientation angle and corresponding to the second axis of the extended thumb as shown in said second image;

(k) determining the three-dimensional position coordinates of the extended thumb second end portion by identifying coordinates of a second virtual intersection of said third and fourth virtual lines;

(l) determining a third parameter vector representative of a third linear projection of the thumb second axis in said first image in accordance with said third set of pixel coordinates and said third orientation angle, and determining a fourth parameter vector representative of a fourth linear projection of the thumb second axis in said second image in accordance with said fourth set of pixel coordinates and said fourth orientation angle;

(m) defining a third virtual plane along said third parameter vector, and defining a fourth virtual plane along said fourth parameter vector; and

(n) determining the three-dimensional orientation angles of the extended thumb by identifying orientation parameters of a second intersecting line defined by an intersection of said third and fourth virtual planes.

2. The method of claim 1, further comprising the step of:

(o) prior to said step (a), calibrating the first and second video acquisition devices.

3. The method of claim 1, wherein said orientation angles comprise azimuth and elevation angles.

4. The method of claim 1, wherein said step (b) further comprises the steps of:

(p) defining a plurality of regions in each of said first and second images, with each defined region being represented by a corresponding boundary of pixels;

(q) screening each of the corresponding boundaries of the defined regions to identify one of said defined regions, in each of said first and second images, containing an image of the user'"'"'s hand;

(r) locating extreme concave and convex curvature values on the corresponding boundary of said identified one region, in each of said first and second images, and defining said concave curvature values as valleys and said convex curvature values as peaks; and

(s) determining a count of the defined peaks and valleys and correlating the count with prestored data representative of said pointing gesture to determine whether both first and second images correspond to said pointing gesture.

5. The method of claim 4, wherein each of the corresponding boundaries has a length, and wherein said step (q) comprises comparing the boundary lengths to a preset range and selecting a boundary having a length within said preset range as the boundary of said identified one region in each of said first and second images.

6. The method of claim 4, wherein said step (s) comprises detecting the index finger in said defined region containing an image of a hand in said first and second images.

7. The method of claim 4, wherein at said step (q) said boundaries are screened at a predefined sampling rate, and wherein at said step (r) said extreme concave and convex curvature values are located by comparing said curvature values to a curvature threshold, further comprising the step of:

(t) when the thumb is determined not to be extended at said step (B),

(4) increasing said predefined sampling rate by a first predetermined quantity,

(5) lowering said curvature threshold by a second predetermined quantity, and

(6) repeating said steps (q) and (r) in accordance with said increased sampling rate and said decreased curvature threshold.

8. A system for tracking, in a real world scene, three-dimensional position coordinates and orientation angles of a user'"'"'s hand and of the user'"'"'s extended index finger having a first axis and of an extended thumb having a second axis, said the system comprising:

a first video acquisition device and a second video acquisition device for monitoring, from different positions, an identification zone defined in the real world scene, and

a computer connected to said first and second video acquisition devices and operable for;

acquiring a first image of the user'"'"'s hand from the first video acquisition device and simultaneously acquiring a second image of the user'"'"'s hand from the second video acquisition device when the user'"'"'s hand is disposed within the identification zone;

analyzing said first and second images to determine whether both said first and second images correspond to a pointing gesture of the user'"'"'s hand wherein at least the index finger is extended, and

when both said first and second images are determined to correspond to the pointing gesture of the user'"'"'s hand;

identifying a first end portion of the extended index finger on each of said first and second images;

determining whether the thumb is extended, and when the thumb is determined to be extended, identifying a second end portion of the extended thumb on each of said first and second images;

when only one of said first and second images is determined to correspond to the pointing gesture of the user'"'"'s hand, acquiring a first image of the user'"'"'s hand from the first video acquisition device and simultaneously acquiring a second image of the user'"'"'s hand from the second video acquisition device; and

when both said first and second images are determined to not correspond to the pointing gesture of the user'"'"'s hand, acquiring a first image of the user'"'"'s hand from the first video acquisition device and simultaneously acquiring a second image of the user'"'"'s hand from the second video acquisition device;

determining a first set of pixel coordinates and a first orientation angle for the first end portion of the extended index finger from said first image and determining a second set of pixel coordinates and a second orientation angle for the first end portion of the extended index finger from said second image;

defining a first virtual line in the real world scene in accordance with said first set of pixel coordinates and said first orientation angle and corresponding to the first axis of the extended index finger as shown in said first image, and defining a second virtual line in the real world scene in accordance with said second set of pixel coordinates and said second orientation angle and corresponding to the second axis of the extended index finger as shown in said second image;

determining the three-dimensional position coordinates of the extended index finger first end portion by identifying coordinates of a first virtual intersection of said first and second virtual lines;

determining a first parameter vector representative of a first linear projection of the index finger first axis in said first image in accordance with said first set of pixel coordinates and said first orientation angle, and determining a second parameter vector representative of a second linear projection of the index finger first axis in said second image in accordance with said second set of pixel coordinates and said second orientation angle;

defining a first virtual plane along said first parameter vector, and defining a second virtual plane along said second parameter vector;

determining the three-dimensional orientation angles of the extended index finger by identifying orientation parameters of a first intersecting line defined by an intersection of said first and said second virtual planes;

when the thumb is determined to be extended, determining a third set of pixel coordinates and a third orientation angle for the second end portion of the extended thumb from said first image and determining a fourth set of pixel coordinates and a fourth orientation angle for the second end portion of the extended thumb from said second image;

defining a third virtual line in the real world scene in accordance with said third set of pixel coordinates and said third orientation angle and corresponding to the second axis of the extended thumb as shown in said second image, and defining a fourth virtual line in the real world scene in accordance with said fourth set of pixel coordinates and said fourth orientation angle and corresponding to the second axis of the extended thumb as shown in said second image;

determining the three-dimensional position coordinates of the extended thumb second end portion by identifying coordinates of a second virtual intersection of said third and fourth virtual lines;

determining a third parameter vector representative of a third linear projection of the thumb second axis in said second image in accordance with said third set of pixel coordinates and said third orientation angle, and determining a fourth parameter vector representative of a fourth linear projection of the thumb second axis in said second image in accordance with said fourth set of pixel coordinates and said fourth orientation angle;

defining a third virtual plane along said third parameter vector, and defining a fourth virtual plane along said fourth parameter vector; and

determining the three-dimensional orientation angles of the extended thumb by identifying orientation parameters of a second intersecting line defined by an intersection of said third and said fourth virtual planes.

9. The system of claim 8, wherein said orientation angles comprise azimuth and elevation angles.