Apparatus, method and computer program for recognizing a gesture in a picture, and apparatus, method and computer program for controlling a device

• US 8,570,383 B2
• Filed: 04/22/2011
• Issued: 10/29/2013
• Est. Priority Date: 10/23/2008
• Status: Active Grant

• Current Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Original Assignee: Fraunhofer Gesellschaft Zur Foerderung Der Angewandten Forsching E.V.
• Inventors: Klefenz, Frank, Hardzeyeu, Valiantsin
• Primary Examiner(s): Tran, Nhan T
• Assistant Examiner(s): CAMARGO, MARLY S.B.
• Application Number:
  US13/092,184
• Publication Number:
  US 20110234840A1
• Time in Patent Office: 921 Days
• Field of Search: 348/222.1, 348/208.14, 348/77, 348/169, 348/94, 348/26, 348/252, 348/246, 348/597, 382/194, 382/195, 382/190, 382/203, 382/209, 382/197, 382/199, 382/266, 382/269, 382/281, 382/224, 345/181, 345/427, 345/604, 345/645
• US Class Current: 348/207.99
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An apparatus for recognizing gestures in a picture includes a Hough transformer configured to identify elements in the picture or in a pre-processed version of the picture as identified gesture elements and to obtain information about the identified gesture elements.

The apparatus further includes a gesture description creator configured to obtain a gesture description while using the information about the identified gesture elements.

Moreover, the apparatus includes a gesture classifier configured to compare the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them. The gesture classifier is configured to provide, as the result of the comparison, a gesture code of a recognized gesture.

Claims

1. An apparatus for recognizing gestures in a picture, using hardware, a computer system, or a combination of hardware and a computer system, comprising:

a Hough transformer configured to identify arcs in the picture or in a pre-processed version of the picture as identified gesture elements so as to acquire information about the identified gesture elements;

a gesture description creator configured to acquire a gesture description while using the information about the identified gesture elements; and

a gesture classifier configured to compare the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them so as to provide, as the result of the comparison, a gesture code of a recognized gesture;

wherein the Hough transformer is configured to identify a plurality of straight-line segments extending through the picture in various directions as identified gesture elements;

wherein the Hough transformer is configured to provide information about a location, a length or a direction of an identified straight-line segment, and

wherein the gesture description creator is configured to use the information, provided by the Hough transformer, about the identified straight-line segments for creating the gesture description;

wherein the gesture description creator is configured to acquire as the gesture description a description of a gesture describing the gesture as an ordered listing of parameters of identified gesture elements.

2. The apparatus as claimed in claim 1, the apparatus comprising an area parameter determiner configured to determine an area parameter of a gesture area, and

the gesture description creator being configured to acquire the gesture description while using the area parameter.

3. The apparatus as claimed in claim 2, wherein the area parameter describes a property of the area independently of an area contour.

4. The apparatus as claimed in claim 2, wherein the area parameter determiner is configured to acquire, as the area parameter, information about an area centroid of the gesture area, or

information about an expansion of the gesture area,

or information about a momentum of the gesture area

or information about an eccentricity of the gesture area.

5. The apparatus as claimed in claim 2, wherein the gesture description creator is configured to provide as the gesture description a set of parameters describing contour sections of the gesture area, and

wherein the gesture description creator is configured to normalize the gesture description while using the area parameter.

6. The apparatus as claimed in claim 2, wherein the area parameter determiner is configured to acquire as the area parameter information about a location of an area centroid of the gesture area, and

wherein the gesture description creator is configured to provide the gesture description such that location parameters of the contour sections of the gesture area are related to the location of the area centroid of the gesture area.

7. The apparatus as claimed in claim 2, wherein the area parameter determiner is configured to acquire as the area parameter information about a dimension of the gesture area, and

wherein the gesture description creator is configured to provide the gesture description such that location parameters or dimension parameters of the contour sections are normalized, in terms of size, on the basis of the information about the dimensions of the gesture area.

8. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to order the gesture description in such a manner

that the ordered identified gesture elements are ordered in accordance with their sequence along a continuous contour line of a gesture area, or

that the ordered identified gesture elements describe a continuous contour line of the gesture area.

9. The apparatus as claimed in claim 1, wherein the Hough transformer is configured to provide information about a location, an orientation, an arc length, a radius of curvature, or an angle of curvature of the identified arcs;

and

wherein the gesture description creator is configured to use the information, provided by the Hough transformer, about the location, the orientation, the arc length, the radius of curvature, or the angle of curvature of the identified arcs for creating the gesture description.

10. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to create the gesture description such that the gesture description comprises a description of a relative location of circular arcs or elliptical arcs that belong to a contour line of a gesture area.

11. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to comprise information, provided by the Hough transformer, about an orientation of the respective circular arc or elliptical arc into the gesture description for identified circular arcs or elliptical arcs.

12. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to comprise information, provided by the Hough transformer, about a position of an extreme point of the identified circular arc or of the identified elliptical arc into the gesture description for identified circular arcs or elliptical arcs.

13. The apparatus as claimed in claim 1, wherein the Hough transformer is configured to identify only such circular arcs or elliptical arcs whose radii of curvature are smaller, in magnitude, than a predefined maximally admissible radius of curvature.

14. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to create a description of the gesture by stitching together selected adjacent identified gesture elements, and

wherein the gesture description creator is configured to select the selected gesture elements used for the description of the gesture from a totality of identified gesture elements, so that the selected gesture elements describe a continuous line shape from an initial point to an end point.

15. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to create, on the basis of the identified gesture elements, a feature vector describing successive sections of a contour line of a gesture area.

16. The apparatus as claimed in claim 1, wherein the gesture classifier comprises a database comparator configured to compare a feature vector comprising the information of the gesture description to a plurality of comparative feature vectors associated with comparative gestures so as to acquire a measure of differences between the feature vector and the comparative feature vectors and to determine, on the basis of the measure of the differences, a gesture code belonging to the feature vector.

17. The apparatus as claimed in claim 1, the apparatus comprising an edge recognizer configured to recognize edges in the picture and to create, on the basis of the picture, an edge picture as the pre-processed version of the picture.

18. The apparatus as claimed in claim 1, the apparatus being configured to identify pixels whose color values are in a predefined range as pixels belonging to a gesture area.

19. The apparatus as claimed in claim 1, the apparatus comprising a picture detail selector configured to identify, in the picture, a picture detail that comprises a contiguous gesture area, and

wherein the Hough transformer is configured to process a pre-processed version of the picture that corresponds to the picture detail.

20. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to provide as the gesture description an ordered set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to compare the ordered set of parameters to a plurality of comparative parameter sets associated with comparative gestures, so as to acquire information about differences between the ordered set of parameters and the comparative parameter sets, and

wherein the gesture classifier is configured to compare one or more cyclically rotated-through versions of the ordered set of parameters to the plurality of comparative parameter sets so as to acquire information about differences between one or more cyclically rotated-through versions of the ordered set of parameters and the comparative parameter sets, and

wherein the gesture classifier is configured to determine, on the basis of the information about the differences, a gesture code belonging to the ordered set of parameters or belonging to a rotated-through version of the ordered set of parameters.

21. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to provide as the gesture description an ordered set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to compare the ordered set of parameters to a plurality of comparative parameter sets that are associated with comparative gestures, so as to acquire information about differences between the ordered set of parameters and the comparative parameter sets, and

wherein the gesture classifier is configured to compare one or more scaled versions of the ordered set of parameters to the plurality of comparative parameter sets so as to acquire information about differences between the at least one scaled version of the ordered set of parameters and the comparative parameter sets, and

wherein the gesture classifier is configured to determine, on the basis of the information about the differences, a gesture code belonging to the ordered set of parameters or belonging to a scaled version of the ordered set of parameters.

22. The apparatus as claimed in claim 21, wherein the gesture classifier is configured to acquire the one or more scaled versions of the ordered set of parameters such that the one or more scaled versions describe a contour of a scaled gesture area that is enlarged or reduced in size as compared to a gesture area actually comprised by the picture.

23. The apparatus as claimed in claim 21, wherein the gesture classifier is configured to scale radius-of-curvature information that is provided by the Hough transformer and is part of the ordered set of parameters, so as to acquire the one or more scaled versions of the ordered set.

24. The apparatus as claimed in claim 21, wherein the gesture classifier is configured to scale length information of straight lines that are part of the ordered set of parameters so as to acquire the one or more scaled versions of the ordered set of parameters.

25. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to provide as the gesture description a set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to determine, by means of a physical model of a body part forming the gesture, a probability with which the posture, described by the set of parameters, of individual elements of the body part is assumed, and

wherein the gesture classifier is configured to take into account the probability when determining the gesture code.

26. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to provide as the gesture description a set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to detect whether the set of parameter values violates a predefined physical location condition, and

to suppress provision of the gesture code or to signal an error if the set of parameter values violates the predefined location condition.

27. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to provide as the gesture description a set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to determine a current gesture code describing a current gesture state in dependence on a previous gesture code that is determined on the basis of a previous picture and describes a previous gesture state.

28. The apparatus as claimed in claim 27, wherein the gesture classifier is configured to evaluate, when determining the current gesture code, state transition admissibility information describing which current gesture states may be assumed on the basis of a previous gesture state.

29. The apparatus as claimed in claim 28, wherein the gesture classifier is configured to suppress such transitions from a previous gesture state to a current gesture state that are described as inadmissible by the state transition admissibility information.

30. The apparatus as claimed in claim 1, wherein the gesture description creator is configured to acquire information about a rotation of the gesture area, and

wherein the gesture classifier is configured to take into account the information about the rotation of the gesture area when determining the gesture code.

31. The apparatus as claimed in claim 30, wherein the gesture classifier is configured to distinguish gestures in dependence on the information about the rotation of the gesture area.

32. The apparatus as claimed in claim 30, wherein the gesture classifier is configured to determine the information about the rotation of the gesture area on the basis of a temporal development of location parameters of the identified gesture elements.

33. The apparatus as claimed in claim 30, wherein the gesture classifier is configured to acquire the information about the rotation of the gesture area while using area parameters describing momenta of the gesture area with regard to different directions.

34. The apparatus as claimed in claim 1, the apparatus being configured to recognize a hand gesture,

the gesture description creator being configured to create the gesture description such that the gesture description comprises a description of arcs of different orientations describing fingertips and interstices between fingers.

37. A camera system comprising:

a camera configured to provide a camera signal describing a camera picture taken by the camera;

an apparatus for recognizing gestures in a picture in accordance with claim 1, the apparatus for recognizing gestures being configured to receive the camera picture and to acquire the gesture code on the basis thereof; and

a camera controller configured to set an alignment, focussing or function of the camera in dependence on the gesture code of the gesture identified in the camera picture.

38. A device configuration, comprising:

a device comprising a device controller;

a camera configured to provide a camera signal describing a camera picture taken by the camera; and

an apparatus for recognizing gestures in a picture in accordance with claim 1, the apparatus for recognizing gestures being configured to receive the camera picture and to acquire the gesture code on the basis thereof;

the device controller being configured to control a function of the device in dependence on the gesture code.

35. A method for recognizing gestures in a picture, comprising:

performing at least one Hough transform for identifying arcs in the picture or in a pre-processed version of the picture as identified gesture elements so as to acquire information about the identified gesture elements,

and to identify a plurality of straight-line segments extending through the picture in various directions as identified gesture elements to provide information about a location, a length or a direction of an identified straight-line segment;

creating a gesture description while using the information about the identified gesture elements, so that the gesture description describes the gesture as an ordered listing of parameters of identified gesture elements,

the information about the identified straight-line segments that is provided by the Hough transform being used for creating the gesture description; and

comparing the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them so as to provide a gesture code of a recognized gesture as the result of the comparison; and

the method for recognizing gestures in a picture is implemented using hardware, a computer system, or a combination of hardware and a computer system.

36. A non-transitory computer readable medium comprising a computer program for performing, when the computer program is run on a computer, a method for recognizing gestures in a picture, said method comprising:

performing at least one Hough transform for identifying arcs in the picture or in a pre-processed version of the picture as identified gesture elements so as to acquire information about the identified gesture elements,

and to identify a plurality of straight-line segments extending through the picture in various directions as identified gesture elements to provide information about a location, a length or a direction of an identified straight-line segment;

creating a gesture description while using the information about the identified gesture elements,

so that the gesture description describes the gesture as an ordered listing of parameters of identified gesture elements,

the information about the identified straight-line segments that is provided by the Hough transform being used for creating the gesture description; and

comparing the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them so as to provide a gesture code of a recognized gesture as the result of the comparison.

39. An apparatus for recognizing gestures in a picture, using hardware, a computer system, or a combination of hardware and a computer system, comprising:

a Hough transformer configured to identify arcs in the picture or in a pre-processed version of the picture as identified gesture elements so as to acquire information about the identified gesture elements;

a gesture description creator configured to acquire a gesture description while using the information about the identified gesture elements; and

a gesture classifier configured to compare the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them so as to provide, as the result of the comparison, a gesture code of a recognized gesture;

wherein the gesture description creator is configured to describe as the gesture description an ordered set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to compare the ordered set of parameters to a plurality of comparative parameter sets associated with comparative gestures, so as to acquire information about differences between the ordered set of parameters and the comparative parameter sets, and wherein the gesture classifier is configured to compare one or more cyclically rotated-through versions of the ordered set of parameters to the plurality of comparative parameter sets so as to acquire information about differences between one or more cyclically rotated-through versions of the ordered set of parameters and the comparative parameter sets, and

wherein the gesture classifier is configured to determine, on the basis of the information about the differences, a gesture code belonging to the ordered set of parameters or belonging to a rotated-through version of the ordered set of parameters.

40. An apparatus for recognizing gestures in a picture, using hardware, a computer system, or a combination of hardware and a computer system, comprising:

a Hough transformer configured to identify arcs in the picture or in a pre-processed version of the picture as identified gesture elements so as to acquire information about the identified gesture elements;

a gesture description creator configured to acquire a gesture description while using the information about the identified gesture elements; and

a gesture classifier configured to compare the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them so as to provide, as the result of the comparison, a gesture code of a recognized gesture;

wherein the gesture description creator is configured to describe as the gesture description an ordered set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to compare the ordered set of parameters to a plurality of comparative parameter sets associated with comparative gestures, so as to acquire information about differences between the ordered set of parameters and the comparative parameter sets, and

wherein the gesture classifier is configured to compare one or more scaled versions of the ordered set of parameters to the plurality of comparative parameter sets so as to acquire information about differences between the at least one scaled version of the ordered set of parameters and the comparative parameter sets, and

wherein the gesture classifier is configured to determine, on the basis of the information about the differences, a gesture code belonging to the ordered set of parameters or belonging to a scaled version of the ordered set of parameters.

41. An apparatus for recognizing gestures in a picture, using hardware, a computer system, or a combination of hardware and a computer system, comprising:

a Hough transformer configured to identify arcs in the picture or in a pre-processed version of the picture as identified gesture elements so as to acquire information about the identified gesture elements;

a gesture description creator configured to acquire a gesture description while using the information about the identified gesture elements; and

a gesture classifier configured to compare the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them so as to provide, as the result of the comparison, a gesture code of a recognized gesture;

wherein the gesture description creator is configured to describe as the gesture description a set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to detect whether the set of parameter values violates a predefined physical location condition, and

to suppress provision of the gesture code or to signal an error if the set of parameter values violates the predefined location condition.

42. An apparatus for recognizing gestures in a picture, using hardware, a computer system, or a combination of hardware and a computer system, comprising:

a Hough transformer configured to identify arcs in the picture or in a pre-processed version of the picture as identified gesture elements so as to acquire information about the identified gesture elements;

a gesture description creator configured to acquire a gesture description while using the information about the identified gesture elements; and

a gesture classifier configured to compare the gesture description to a plurality of comparative gesture descriptions having gesture codes associated with them so as to provide, as the result of the comparison, a gesture code of a recognized gesture;

wherein the gesture description creator is configured to provide as the gesture description a set of parameters describing contour sections of a gesture area, and

wherein the gesture classifier is configured to determine a current gesture code describing a current gesture state in dependence on a previous gesture code that is determined on the basis of a previous picture and describes a previous gesture state.