Gesture-based input interface system with shadow detection

• US 6,624,833 B1
• Filed: 04/17/2000
• Issued: 09/23/2003
• Est. Priority Date: 04/17/2000
• Status: Expired due to Term

• Current Assignee: WSOU Investments, LLC (WSOU Holdings, LLC)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Kumar, Senthil, Segen, Jakub
• Primary Examiner(s): Cabeca, John
• Assistant Examiner(s): BASOM, BLAINE T
• Application Number:
  US09/551,041
• Time in Patent Office: 1,254 Days
• Field of Search: 345/863, 345/156, 345/157, 345/158
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

An input interface system provides gesture-based user control of an application running on a computer by classification of user gestures in image signals. A given one of the image signals is processed to determine if it contains one of a number of designated user gestures, e.g., a point gesture, a reach gesture and a click gesture, each of the gestures being translatable to a particular control signal for controlling the application. If the image signal is determined to contain a point gesture, further processing is performed to determine position and orientation information for a pointing finger of a hand of the user and its corresponding shadow. The position and orientation information for the pointing finger and its shadow are then utilized to generate a three-dimensional pose estimate for the pointing figure in the point gesture. For example, the three-dimensional pose estimate may be in the form of a set of five parameters (X, Y, Z, α, ε), where (X, Y, Z) denotes the position of a tip of the pointing finger in three-dimensional space, and (α, ε) denotes the respective azimuth and elevation angles of an axis of the pointing finger. The point gesture can thus be used to provide user control in virtual flight simulators, graphical editors, video games and other applications.

Claims

1. A method of providing gesture-based user control of an application running on a computer, he method comprising the steps of:

receiving at least one image signal; and

processing the at least one image signal to determine if the image contains one of a plurality of designated user gestures, each of the gestures being translatable to a particular control signal for controlling the application, the processing step for a given one of the designated user gestures determining position and orientation information for a pointing finger of a hand of the user and its corresponding shadow, the position and orientation information for the pointing finger and its shadow being utilized to generate a three-dimensional pose estimate for the pointing figure in the given gesture.

2. The method of claim 1 wherein the plurality of designated user gestures comprises at least a point gesture, a reach gesture and a click gesture.

3. The method of claim 1 wherein the pointing finger comprises an index finger of the user.

4. The method of claim 1 wherein the three-dimensional pose estimate comprises a set of five parameters (X, Y, Z, α

, ε

), where (X, Y, Z) denotes the position of a tip of the pointing finger in three-dimensional space, and (α

, ε

) denotes the respective azimuth and elevation angles of an axis of the pointing finger.

5. The method of claim 1 wherein the position and orientation for the pointing finger comprises a pair of two-dimensional poses, one representing a peak in the image signal corresponding to, the pointing finger and the other representing a peak in the image signal corresponding to the shadow of the pointing finger.

6. The method of claim 5 wherein the pair of two-dimensional poses is of the form {(x,y,θ

), (x_s,y_s,θ

_s)}, where (x,y) and θ

denote the position of a tip and orientation, respectively, of the pointing finger in two-dimensional space, and (x_s,y_s) and θ

_s denote the position of a tip and orientation, respectively, of the shadow of the pointing finger in two-dimensional space.

7. The method of claim 1 wherein the processing step further includes performing an initial classification of a given image as containing one of a point gesture, a reach gesture and a ground gesture based on extraction of local features, followed by a finite state classification which classifies the given image as containing one of the point gesture, a click gesture and a ground gesture, the position and orientation information for the pointing finger and its corresponding shadow being determined only if the image is classified as containing the point gesture.

8. The method of claim 7 wherein the local features comprise peaks and valleys in a boundary of a connected region representative of the hand in the given image, the peaks corresponding to points of curvature maxima where the boundary is locally convex, and the valleys corresponding to points of curvature minima where the boundary is locally concave, the initial classification being made based on the number of peaks and the number of valleys.

9. The method of claim 7 wherein the result of the finite state classification for the given image is a function of a result of the finite state classification for a previous image and a result of the initial classification for the given image.

10. The method of claim 7 wherein the finite state classification determines if the initial classification is a point gesture, and if so attempts to determine the position and orientation of the pointing finger and its corresponding shadow in the image.

11. The method of claim 1 wherein the generation of the three-dimensional pose estimate for the pointing finger in the given gesture utilizes calibration information associated with a light source which is utilized to produce the shadow of the pointing finger.

12. An apparatus for providing gesture-based user control of an application running on a computer, the apparatus comprising:

a processing device associated with the computer and operative to process at least one image signal to determine if the image contains one of a plurality of designated user gestures, each of the gestures being translatable to a particular control signal for controlling the application, the processing step for a given one of the designated user gestures determining position and orientation information for a pointing finger of a hand of the user and its corresponding shadow, the position and orientation information for the pointing finger and its shadow being utilized to generate a three-dimensional pose estimate for the pointing figure in the given gesture.

13. An input interface system for providing gesture-based user control of an application, the system comprising:

at least one camera;

a light source arranged so as to create a shadow of a user hand within a field of view of the camera; and

a computer coupled to the camera and operative to process at least one image signal received from the camera to determine if the image contains one of a plurality of designated user gestures, each of the gestures being translatable to a particular control signal for controlling the application, the processing step for a given one of the designated user gestures determining position and orientation information for a pointing finger of a hand of the user and its corresponding shadow, the position and orientation information for the pointing finger and its shadow being utilized to generate a three-dimensional pose estimate for the pointing figure in the given gesture.

14. An article of manufacture containing one or more software programs which, when executed in a processing device, cause the processing device to perform the step of processing at least ode image signal to determine if the image contains one of a plurality of designated user gestures, each of the gestures being translatable to a particular control signal for controlling the application, the processing step for a given one of the designated user gestures determining position and orientation information for a pointing finger of a hand of the user and its corresponding shadow, the position and orientation information for the pointing finger and its shadow being utilized to generate a three-dimensional pose estimate for the pointing figure in the given gesture.