Gesture recognition systems and related methods

• US 9,417,700 B2
• Filed: 05/20/2010
• Issued: 08/16/2016
• Est. Priority Date: 05/21/2009
• Status: Active Grant

• Current Assignee: Golden Edge Holding Corporation
• Original Assignee: Edge 3 Technologies Incorporated
• Inventors: El Dokor, Tarek A., King, Joshua T.
• Primary Examiner(s): Awad, Amr
• Assistant Examiner(s): BRAY, STEPHEN A
• Application Number:
  US12/784,123
• Publication Number:
  US 20100295783A1
• Time in Patent Office: 2,280 Days
• Field of Search: 345156-158, 382/103, 382/173
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

In one embodiment of the invention, the a method includes the steps of receiving one or more raw frames from one or more cameras, each of the one or more raw frames representing a time sequence of images, determining one or more regions of the one or more received raw frames that comprise highly textured regions, segmenting the one or more determined highly textured regions in accordance textured features thereof to determine one or more segments thereof, determining one or more regions of the one or more received raw frames that comprise other than highly textured regions, and segmenting the one or more determined other than highly textured regions in accordance with color thereof to determine one or more segments thereof. One or more of the segments are then tracked through the one or more raw frames representing the time sequence of images.

Claims

1. A method for performing gesture recognition, comprising the steps of:

receiving one or more raw frames from one or more cameras, each of the one or more raw frames representing a time sequence of images;

segmenting the one or more received raw frames in near-real time by a computer processor by;

first determining by the computer processor one or more regions of the one or more received raw frames that comprise highly textured regions;

second segmenting by the computer processor the one or more determined highly textured regions in accordance with textured features thereof to determine one or more segments thereof;

third determining by the computer processor the remaining of one or more regions of the one or more received raw frames that comprise other than highly textured regions; and

fourth segmenting by the computer processor the remaining one or more of the determined other than highly textured regions in accordance with color thereof to determine one or more segments thereof;

after segmenting the one or more received frames, determining by the computer processor a depth map in accordance with the segmented one or more raw frames; and

tracking one or more of the segments through the one or more raw frames in accordance with the determined depth map representing the time sequence of images.

2. The method of claim 1, wherein the segmenting further comprises the step of:

before determining one or more regions of the one or more received raw frames that comprise highly textured regions, performing by the computer processor an initial motion-based segmentation of the one or more received raw frames.

3. The method of claim 1, wherein the segmenting further comprises the steps of:

determining by the computer processor position changes of one or more portions between the received raw frames indicating motion of a portion thereof; and

segmenting the received raw frames in accordance with these determined position changes.

4. The method of claim 1, wherein raw images are received from two cameras, and further comprising the step of tracking one or more of the segments through corresponding ones of the one or more raw frames received from each of the two cameras.

5. The method of claim 4, further comprising the step of determining by the computer processor one or more disparities between the corresponding one of the one or more frames in accordance with the segments tracked therein.

6. The method of claim 5, wherein the one or more disparities are determined by the computer processor in accordance with the application of disparity space decomposition.

7. The method of claim 1, further comprising the steps of:

determining by the computer processor a particular segment to be tracked;

determining by the computer processor whether the particular segment is providing a static gesture or a dynamic gesture in accordance with a determination of change of position of the particular segment in the time sequence of images.

8. The method of claim 7, further comprising the step of applying a deep belief network for gesture recognition if it is determined that the particular segment is providing a static gesture.

9. The method of claim 7, further comprising the step of applying a disparity space decomposition procedure for gesture recognition if it is determined that the particular segment is providing a dynamic gesture.

10. The method of claim 7, further comprising the steps of:

associating the gesture with a predetermined action; and

performing the action upon recognition of the gesture.

11. The method of claim 1, wherein:

segmenting by the computer processor the one or more determined highly textured regions in accordance with textured features thereof to determine one or more segments thereof is performed in accordance with a morphological analysis of the area employing a convolution kernel to perform texture feature extraction; and

segmenting by the computer processor the one or more determined other than highly textured regions in accordance with color thereof to determine one or more segments thereof is performed in accordance with one of block-based medial filtering and color-based clustering, and a stereo correspondence algorithm to determine the pixel depths in the other than highly textured regions.

12. A system for performing gesture recognition, comprising:

an input port for receiving one or more raw frames from one or more cameras, each of the one or more raw frames representing a time sequence of images; and

a processor for segmenting in near-real time without additional user input the one or more received raw frames by first determining one or more regions of the one or more received raw frames that comprise highly textured regions, second segmenting the one or more determined highly textured regions in accordance with textured features thereof to determine one or more segments thereof, third determining the remaining of one or more regions of the one or more received raw frames that comprise other than highly textured regions, and fourth segmenting the remaining one or more of the determined other than highly textured regions in accordance with color thereof to determine one or more segments thereof, after segmenting the one or more received frames, generating a depth map in accordance with the segmented one or more received raw frames, and tracking one or more of the segments through the one or more raw frames representing the time sequence of images.

13. The system of claim 12, wherein the processor is further adapted to, before determining one or more regions of the one or more received raw frames that comprise highly textured regions, perform in near-real time an initial motion-based segmentation of the one or more received raw frames.

14. The system of claim 12, further comprising two cameras for providing the raw images.

15. The method of claim 12, wherein the processor further performs:

segmenting by the computer processor the one or more determined highly textured regions in accordance with textured features thereof to determine one or more segments thereof is performed in accordance with a morphological analysis of the area employing a convolution kernel to perform texture feature extraction; and

segmenting by the computer processor the one or more determined other than highly textured regions in accordance with color thereof to determine one or more segments thereof is performed in accordance with one of block-based medial filtering and color-based clustering, and a stereo correspondence algorithm to determine the pixel depths in the other than highly textured regions.