Visual language for human computer interfaces

• US 9,524,028 B2
• Filed: 03/14/2013
• Issued: 12/20/2016
• Est. Priority Date: 03/08/2013
• Status: Active Grant

• Current Assignee: FastVDO LLC
• Original Assignee: FastVDO LLC
• Inventors: Dai, Wei, Krishnan, Madhu Peringassery, Topiwala, Pankaj
• Primary Examiner(s): Ghebretinsae, Temesghen
• Assistant Examiner(s): Ketema, Benyam
• Application Number:
  US13/830,787
• Publication Number:
  US 20140253429A1
• Time in Patent Office: 1,377 Days
• Field of Search: 345/156
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

Embodiments of the invention recognize human visual gestures, as captured by image and video sensors, to develop a visual language for a variety of human computer interfaces. One embodiment provides a method for recognizing a hand gesture positioned by a user hand. The method includes steps of capturing a digital color image of a user hand against a background, applying a general parametric model to the digital color image of the user hand to generate a specific parametric template of the user hand, receiving a second digital image of the user hand positioned to represent a hand gesture, detecting a hand contour of the hand gesture based at least in part on the specific parametric template of the user hand, and recognizing the hand gesture based at least in part on the detected hand contour. Other embodiments include recognizing hand gestures, facial gestures or body gestures captured in a video.

Claims

1. A computer-implemented method for recognizing hand gestures, the method comprising:

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

receiving a digital color image of a user hand against a background, the digital color image captured using a 2-D digital image capturing device, the digital color image being represented by pixels in a first color space,

selecting a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

obtaining a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand, obtaining the contour of the user hand comprising;

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points; and

detecting a hand gesture based on the contour of the user hand, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

2. The method of claim 1, wherein the selected set of pixels are in a predefined hand outline.

3. The method of claim 1, wherein the selected set of pixels are in a predefined region of interest.

4. The method of claim 1, wherein a specific parametric template of the specific parametric templates is obtained by:

applying a color model to the selected set of pixels of the digital color image of the user hand;

generating a skin map of the user hand; and

generating the specific parametric template based on the generated skin map of the user hand.

5. The method of claim 1, further comprising:

applying two independent color space models to the digital color image of the user hand;

generating two skin maps of the user hand using the pixels in the digital color image of the user hand; and

combining the two skin maps to generate an updated skin map of the user hand.

6. The method of claim 5, wherein a first color space model of the two independent color space models is selected from a hue-saturation-value color space and the first color space model is represented by hue, saturation and value parameters.

7. The method of claim 5, wherein a second color space model of the two independent color space models is selected from a luma-chroma color space and the second color space model is represented by luma and chroma parameters.

8. The method of claim 5, wherein applying the two independent color space models to the digital color image of the user hand comprises:

for a color space of a color space model chosen from the color spaces of the two independent color space models;

calculating one or more first-order statistics of the color space model in the chosen color space;

calculating one or more first-order statistics of the color space model in a red-green-blue color space;

determining distribution ranges of each parameter of the chosen color space model in the chosen color space; and

determining distribution ranges of each parameter of the chosen color space model in the red-green-blue color space.

9. The method of claim 5, wherein generating the updated skin map of the user hand comprises:

performing a logical inclusive OR operation to the two generated skin maps, the logical inclusive OR operation selecting a pixel of the digital color image of the user hand responsive to the pixel being in one of the two generated skin maps; and

generating the updated skin map of the user hand based on the logical inclusive OR operation.

10. The method of claim 1, wherein obtaining the contour of the user hand further comprises:

detecting one or more convexity defects of the convex hulls, a convexity defect comprising a start point, a depth point and an end point;

identifying a center of the digital color image of the user hand;

filtering one or more points that do not represent fingertips or finger joints of the user hand; and

refining the convexity defects based on the filtering result to generate a refined polygon approximating the contour of the user hand.

11. The method of claim 10, further comprising:

comparing the refined polygon approximating the contour with a set of known hand gestures; and

determining the hand gesture based on the comparison.

12. The method of claim 10, wherein the set of known hand gestures comprises American Sign Language set of hand gestures.

13. A computer-implemented method for recognizing hand gestures, the method comprising:

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

capturing an input video of a user hand against a background, the input video comprising a plurality of video frames of the user hand captured using a 2-D digital image capturing device, and a video frame of the plurality video frame representing a digital color image of the user hand at a time instance, the digital color image being represented by pixels in a first color space,

selecting a video frame of the plurality of the video frames of the input video as a reference frame,

selecting a set of pixels of the reference frame in the first color space from the pixels of the reference frame within the ROI, the selected set of pixels of the reference frame in the first color space describing a general parametric model associated with the reference frame in the first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

subsequently receiving one or more video frames of the input video of the user hand, wherein the user hand in a subsequently received video frame is positioned to represent a hand gesture;

obtaining contours of the user hand in the subsequent video frames by applying the updated specific parametric template to the subsequent video frames of the user hand, obtaining a contour of the user hand in a video frame comprising;

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points,

obtaining motion vectors of features of the user hand in the subsequent video frames based on the updated specific parametric template; and

detecting a hand gesture based on the contour of the user hand and the motion vectors, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

14. The method of claim 13, wherein the hand gesture is one of a known set of hand gestures.

15. A computer-implemented method for recognizing hand gestures, the method comprising:

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

receiving a digital color image of a user hand against a background, the digital color image captured using a 2-D digital image capturing device, the digital color image being represented by pixels in a first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

obtaining a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand, obtaining the contour of the user hand comprising;

detecting pixels of the set of pixels that represent noise,

regrouping a plurality of isolated regions of the digital color image into one or more segments based on the updated specific parametric template,

selecting a largest segment of the one or more segments to represent the contour of the user hand,

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points; and

detecting a hand gesture based on the contour of the user hand, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

16. A non-transitory computer-readable storage medium storing executable computer program instructions for recognizing hand gestures, the computer programs instructions comprising code for:

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

receiving a digital color image of a user hand against a background, the digital color image captured using a 2-D digital image capturing device, the digital color image being represented by pixels in a first color space,

selecting a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

obtaining a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand, obtaining the contour of the user hand comprising;

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points; and

detecting a hand gesture based on the contour of the user hand, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

17. The computer-readable storage medium of claim 16, wherein the selected set of pixels are in a predefined hand outline.

18. The computer-readable storage medium of claim 16, wherein the selected set of pixels are in a predefined region of interest.

19. The computer-readable storage medium of claim 16, wherein a specific parametric template of the specific parametric templates is obtained by:

applying a color model to the selected set of pixels of the digital color image of the user hand;

generating a skin map of the user hand; and

generating the specific parametric template based on the generated skin map of the user hand.

20. The computer-readable storage medium of claim 16, further comprising computer program instructions for:

applying two independent color space models to the digital color image of the user hand;

generating two skin maps of the user hand using a plurality of the pixels in the digital color image of the user hand; and

combining the two skin maps to generate an updated skin map of the user hand.

21. The computer-readable storage medium of claim 20, wherein a first color space model of the two independent color space models is selected from a hue-saturation-value color space and the first color space model is represented by hue, saturation and value parameters.

22. The computer-readable storage medium of claim 20, wherein a second color space model of the two independent color space models is selected from a luma-chroma color space and the second color space model is represented by luma and chroma parameters.

23. The computer-readable storage medium of claim 20, wherein the computer program instructions for applying the two independent color space models to the digital color image of the user hand comprise computer program instructions for:

for a color space of a color space model chosen from the color spaces of the two independent color space models;

calculating one or more first-order statistics of the color space model in the chosen color space;

calculating one or more first-order statistics of the color space model in a red-green-blue color space;

determining distribution ranges of each parameter of the chosen color space model in the chosen color space; and

determining distribution ranges of each parameter of the chosen color space model in the red-green-blue color space.

24. The computer-readable storage medium of claim 20, wherein the computer program instructions for generating the updated skin map of the user hand comprise computer program instructions for:

performing a logical inclusive OR operation to the two generated skin maps, the logical inclusive OR operation selecting a pixel of the digital color image of the user hand responsive to the pixel being in one of the two generated skin maps; and

generating the updated skin map of the user hand based on the logical inclusive OR operation.

25. The computer-readable storage medium of claim 16, wherein the computer program instructions for obtaining the contour of the user hand further comprise computer program instructions for:

detecting one or more convexity defects of the convex hull, a convexity defect comprising a start point, a depth point and an end point;

identifying a center of the digital color image of the user hand;

filtering one or more points that do not represent fingertips or finger joints of the user hand; and

refining the convexity defects based on the filtering result to generate a refined polygon approximating the contour of the user hand.

26. The computer-readable storage medium of claim 25, further comprising computer program instructions for:

comparing the refined polygon approximating the contour with a set of known hand gestures; and

determining the hand gesture based on the comparison.

27. The computer-readable storage medium of claim 26, wherein the set of known hand gestures comprises American Sign Language set of hand gestures.

28. A non-transitory computer-readable storage medium storing executable computer program instructions for recognizing hand gestures, the computer program instructions comprising code for:

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

capturing an input video of a user hand against a background, the input video comprising a plurality of video frames of the user hand captured using a 2-D digital image capturing device, and a video frame of the plurality video frame representing a digital color image of the user hand at a time instance, the digital color image being represented by pixels in a first color space,

selecting a video frame of the plurality of the video frames of the input video as a reference frame,

selecting a set of pixels of the reference frame in the first color space from the pixels of the reference frame within the ROI, the selected set of pixels of the reference frame in the first color space describing a general parametric model associated with the reference frame in the first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

subsequently receiving one or more video frames of input video of the user hand, wherein the user hand in a subsequently received video frame is positioned to represent a hand gesture;

obtaining contours of the user hand in the subsequent video frames by applying the updated specific parametric template to the subsequent video frames of the user hand, obtaining a contour of the user hand in a video frame comprising;

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points;

obtaining motion vectors features of the user hand in the subsequent video frames based on the updated specific parametric template; and

detecting a hand gesture based on the contour of the user hand and the motion vectors, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

29. The computer-readable storage medium of claim 28, wherein the hand gesture is one of a known set of hand gestures.

30. A non-transitory computer-readable storage medium storing executable computer program instructions for recognizing hand gestures, the computer program instructions comprising code for:

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

receiving a digital color image of a user hand against a background, the digital color image captured using a 2-D digital image capturing device, the digital color image being represented by pixels in a first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

obtaining a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand, obtaining the contour of the user hand comprising;

detecting pixels of the set of pixels that represent noise,

regrouping a plurality of isolated regions of the digital color image into one or more segments based on the updated specific parametric template,

selecting a largest segment of the one or more segments to represent the contour of the user hand,

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points; and

detecting a hand gesture based on the contour of the user hand, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

31. An apparatus for recognizing hand gestures, the apparatus comprising:

a computer processor for performing steps comprising;

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

receiving a digital color image of a user hand against a background, the digital color image captured using a 2-D digital image capturing device, the digital color image being represented by pixels in a first color space,

selecting a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

obtaining a contour of the user hand by applying the updated specific parametric template to subsequent digital images of the user hand, obtaining the contour of the user hand comprising;

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points; and

detecting a hand gesture based on the contour of the user hand, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

32. The apparatus of claim 31, wherein the selected set of pixels are in a predefined hand outline.

33. The apparatus of claim 31, wherein a specific parametric template of the specific parametric templates is obtained by:

applying a color model to the selected set of pixels of the digital color image of the user hand;

generating a skin map of the user hand; and

generating the specific parametric template based on the generated skin map of the user hand.

34. The apparatus of claim 31, further comprising:

applying two independent color space models to the digital color image of the user hand;

generating two skin maps of the user hand using the pixels in the digital color image of the user hand; and

combining the two skin maps to generate an updated skin map of the user hand.

35. The apparatus of claim 34, wherein a first color space model of the two independent color space models is selected from a hue-saturation-value color space and the first color space model is represented by hue, saturation and value parameters.

36. The apparatus of claim 34, wherein a second color space model of the two independent color space models is selected from a luma-chroma color space and the second color space model is represented by luma and chroma parameters.

37. The apparatus of claim 34, wherein applying the two independent color space models to the digital color image of the user hand comprises:

for a color space of a color space model chosen from the color spaces of the two independent color space models;

calculating one or more first-order statistics of the color space model in the chosen color space;

calculating one or more first-order statistics of the color space model in a red-green-blue color space;

determining distribution ranges of each parameter of the chosen color space model in the chosen color space; and

determining distribution ranges of each parameter of the chosen color space model in the red-green-blue color space.

38. The apparatus of claim 34, wherein generating the updated skin map of the user hand comprises:

performing a logical inclusive OR operation to the two generated skin maps, the logical inclusive OR operation selecting a pixel of the digital color image of the user hand responsive to the pixel being in one of the two generated skin maps; and

generating the updated skin map of the user hand based on the logical inclusive OR operation.

39. The apparatus of claim 31, wherein obtaining the contour of the user hand further comprises:

detecting one or more convexity defects of the convex hull, a convexity defect comprising a start point, a depth point and an end point;

identifying a center of the digital color image of the user hand;

filtering one or more points that do not represent fingertips or finger joints of the user hand; and

refining the convexity defects based on the filtering result to generate a refined polygon approximating the contour of the user hand.

40. The apparatus of claim 39, further comprising:

comparing the refined polygon approximating the detected hand contour with a set of known hand gestures; and

determining the hand gesture based on the comparison.

41. The apparatus of claim 40, wherein the set of known hand gestures comprises American Sign Language set of hand gestures.

42. An apparatus for recognizing hand gestures, the apparatus comprising:

a computer processor for performing steps comprising;

performing an adaptive hand detection, performing the adaptive hand detection comprising;

presenting a region of interest (ROI) on a display,

capturing an input video of a user hand against a background, the input video comprising a plurality of video frames of the user hand captured using a 2-D digital image capturing device, and a video frame of the plurality video frame representing a digital color image of the user hand at a time instance, the digital color image being represented in pixels in a first color space,

selecting a video frame of the plurality of the video frames of the input video as a reference frame,

selecting a set of pixels of the reference frame in the first color space from the pixels of the reference frame within the ROI, the selected set of pixels of the reference frame in the first color space describing a general parametric model associated with the reference frame in the first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the user hand in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

subsequently receiving one or more video frames of the input video of the user hand, wherein the user hand in a subsequently received video frame is positioned to represent a hand gesture;

obtaining contours of the user hand in the subsequent video frames by applying the updated specific parametric template to the subsequent video frames of the user hand, obtaining a contour of the user hand in a video frame comprising;

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the user hand based on the points;

obtaining motion vectors of features of the user hand in the subsequent video frames based on the updated specific parametric template; and

detecting a hand gesture based on the contour of the user hand and the motion vectors, detecting the hand gesture comprising analyzing a structure, orientation and motion of the user hand based on the polygonal contour map through a nonlinear classifier to select the hand gesture from a pre-defined vocabulary of gestures.

43. The apparatus of claim 42, wherein the hand gesture is one of a known set of hand gestures.

44. A computer-implemented method for recognizing human visual gestures positioned by a portion of a human body, the method comprising:

performing an adaptive detection of the portion of the human body,

performing the adaptive detection comprising;

presenting a region of interest (ROI) on a display,

receiving a digital color image of the portion of the human body against a background, the digital color image captured using a 2-D digital image capturing device, the digital color image being represented by pixels in a first color space,

selecting a set of pixels of the digital color image in the first color space from the pixels of the digital color image within the ROI, the selected set of pixels of the digital color image in the first color space describing a general parametric model associated with the digital color image in the first color space,

obtaining specific parametric templates in additional color spaces, each specific parametric template comprising a selected set of pixels in the ROI representing the portion of the human body in a corresponding color space of the additional color spaces, the additional color spaces emphasizing chrominance over luminance information, and

obtaining an updated specific parametric template by combining the specific parametric templates in the additional color spaces;

obtaining a contour of the portion of the human body by applying the updated specific parametric template to subsequent digital images of the portion of the human body, obtaining the contour of the portion of the human body comprising;

identifying points corresponding to fingertips, points between fingers, and convex hulls generated by the fingertips in the subsequent digital images based on the updated specific parametric template, and

generating a polygonal contour map of the portion of the human body based on the points; and

detecting a human visual gesture based on the contour, detecting the human visual gesture comprising analyzing a structure, orientation and motion of the portion of the human body based on the polygonal contour map through a nonlinear classifier to select the human visual gesture from a pre-defined vocabulary of gestures.

45. The method of claim 44, wherein the portion of the human body comprises a user face.