Visual Language for Human Computer Interfaces

• US 20170153711A1
• Filed: 11/08/2016
• Published: 06/01/2017
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: FastVDO LLC
• Original Assignee: FastVDO LLC
• Inventors: Dai, Wei, Topiwala, Pankaj, Krishnan, Madhu Peringassery
• Granted Patent: US 10,372,226 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

Embodiments of the invention recognize human visual gestures, as captured by image and video sensors, to develop a visual language for a variety of human computer interfaces. One embodiment provides a method for recognizing a hand gesture positioned by a user hand. The method includes steps of capturing a digital color image of a user hand against a background, applying a general parametric model to the digital color image of the user hand to generate a specific parametric template of the user hand, receiving a second digital image of the user hand positioned to represent a hand gesture, detecting a hand contour of the hand gesture based at least in part on the specific parametric template of the user hand, and recognizing the hand gesture based at least in part on the detected hand contour. Other embodiments include recognizing hand gestures, facial gestures or body gestures captured in a video.

Claims

1. A computer-implemented method for recognizing hand gestures, the method comprising:

presenting a region of interest (ROI) on a display;

receiving a digital color image of a user hand against a background, the digital color image captured using a digital image capturing device, the digital color image being represented by pixels in a first color space;

selecting a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space;

obtaining specific parametric templates in additional color spaces, each specific parametric template in a color space comprising a selected set of pixels within the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information;

combining the specific parametric templates in the additional color spaces to generate an improved specific parametric template;

obtaining a contour of the user hand by applying the improved specific parametric template to subsequent digital images of the user hand; and

detecting a hand gesture based on the contour of the user hand.

2. The computer-implemented method of claim 1, wherein obtaining the contour of the user hand comprises:

identifying points corresponding to finger tips and convex hulls between the finger tips in the subsequent digital images of the user hand based on the improved specific parametric template; and

generating a polygonal contour map of the user hand based on the identified points.

3. The computer-implemented method of claim 2, wherein identifying the points comprises:

detecting one or more convexity defects of the convex hulls, a convexity defect comprising a start point, a depth point and an end point;

identifying a center of the digital color image of the user hand; and

filtering one or more points that do not represent finger tips or finger joints of the user hand.

4. The computer-implemented method of claim 2, wherein detecting the hand gesture comprises analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

5. The computer-implemented method of claim 4, wherein a first color space model of the two independent color space models is selected from a hue-saturation-value color space and the first color space model is represented by hue, saturation and value parameters.

6. The computer-implemented method of claim 5, wherein a second color space model of the two independent color space models is selected from a luma-chroma color space and the second color space model is represented by luma and chroma parameters.

7. The computer-implemented method of claim 4, wherein applying the two independent color space models to the digital color image of the user hand comprises:

for a color space of a color space model chosen from the color spaces of the two independent color space models;

calculating one or more first-order statistics of the color space model in the chosen color space;

calculating one or more first-order statistics of the color space model in a red-green-blue color space;

determining distribution ranges of each parameter of the chosen color space model in the chosen color space; and

determining distribution ranges of each parameter of the chosen color space model in the red-green-blue color space.

8. The computer-implemented method of claim 4, further comprising:

generating an improved skin map of the user hand by;

performing a logical inclusive OR operation to the two generated skin maps, the logical inclusive OR operation selecting a pixel of the digital color image of the user hand responsive to the pixel being in one of the two generated skin maps; and

generating the improved skin map of the user hand based on the logical inclusive OR operation.

9. A non-transitory computer readable medium storing instructions for recognizing hand gestures, the instruction when executed by one or more processors cause the one or more processors to:

present a region of interest (ROI) on a display;

receive a digital color image of a user hand against a background, the digital color image captured using a digital image capturing device, the digital color image being represented by pixels in a first color space;

select a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space;

obtain specific parametric templates in additional color spaces, each specific parametric template in a color space comprising a selected set of pixels within the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information;

combine the specific parametric templates in the additional color spaces to generate an improved specific parametric template;

obtain a contour of the user hand by applying the improved specific parametric template to subsequent digital images of the user hand; and

detect a hand gesture based on the contour of the user hand.

10. A computer implemented method for recognizing a visual gesture, the method comprising:

receiving a first set of images including a part of a human body within a region of interest (ROI), the part of a human body oriented in a first configuration;

registering a flesh tone of the part of the human body within the ROI of the first set of images in a first color space and a second color space;

receiving a second set of images including the part of the human body, the part of the human body oriented in different configurations than the first configuration, the part of the human body oriented in the different configurations representing a visual gesture;

identifying one or more objects in the second set of images corresponding to the part of the human body based on the registered flesh tone in the first color space and the second color space;

obtaining motion vectors of the one or more objects in the second set of images by tracking the one or more objects in the second set of images; and

determining the visual gesture represented by the part of the human body oriented in the different configurations according to the identified one or more objects and the motion vectors.

11. The computer implemented method of claim 10, wherein the part of the human body is a hand, a face, or an entire body.

12. A computer-implemented method for recognizing hand gestures in video imagery, wherein the hand gesture has a static element, a motion element, or a mixture of the two, the method comprising:

pre-registering a user hand, using one or multiple frames of the video;

performing an adaptive hand-detection on subsequent video frames, performing an adaptive hand-detection comprising;

applying at least one of skin color/tone analysis in color spaces and motion estimation, to segment regions or objects, especially a hand, within the video frames,

wherein said skin color/tone analysis in color spaces comprises;

using skin tone analysis in one or more color spaces, to obtain skin map(s),

merging skin maps from the one or more color spaces, and

using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together, and

wherein said motion estimation comprises;

obtaining motion vector fields between two successive frames, typically for blocks within video frames,

tracking motion vectors on subsequent frames,

applying a combination of cluster analysis of motion vectors, and

tracking the evolution of motion vectors on spatial regions and their features; and

recognizing a hand gesture in part by applying at least one of the following four tool groups;

(i) skin color/tone analysis in one or more color spaces, (ii) motion estimation, (iii) morphological operations, and (iv) other image processing tools, on regions or objects, especially a segmented hand, to detect hands, hand contours and their features, as well as any motion of hand parts;

wherein said skin color/tone analysis in one or more color spaces comprises;

using skin tone analysis in one or more color spaces, to obtain skin map(s),

merging skin maps from the one or more color spaces, and

using an adaptive threshold to segment a region or object by detecting skin pixels and grouping them together,

wherein said motion estimation comprises;

obtaining motion vectors between two successive frames, typically for blocks within video frames,

tracking motion vectors on subsequent frames, and

applying any combination of cluster analysis of motion vectors, and tracking the evolution of motion vectors, defined spatial regions and their features, to detect regions or objects within video frames,

wherein said morphological operations include dilations, erosions, opening, and closing operations, and

wherein said other image processing tools include measuring distances, angles, extrema points, convexity, and shape on hand contours.

13. The method of claim 12, wherein the analysis of the evolution of tracked motion vectors includes calculating the spatial position of an object from following the time history of motion vectors, in two or three-dimensional space.

14. The method of claim 12, wherein the adaptive hand-detection processes include at least one of the subprocesses:

position, motion, and shape based processing.

15. The method of claim 12, wherein the hand gesture recognition processes include a position-based gesture recognition process, wherein the position-based gesture recognition process includes analysis of the evolution of tracked motion vectors up to a given video frame.

16. The method of claim 12, wherein the hand gesture recognition processes include a motion-based gesture recognition process, wherein the motion-based gesture recognition process includes analysis of the evolution of motion vectors over multiple frames.

17. The method of claim 12, wherein the hand gesture recognition processes include a shape-based gesture recognition process, wherein the shape-based gesture recognition process includes any combination of skin tone analysis in color spaces, morphological operations, and applying other image processing tools over the plurality of pixels.

18. The method of claim 12, wherein the hand gesture recognition processes include at least two of the subprocesses:

position, motion, and shape based processing.