Method and Apparatus for Identifying a Gesture Based Upon Fusion of Multiple Sensor Signals

• US 20140324888A1
• Filed: 12/09/2011
• Published: 10/30/2014
• Est. Priority Date: 12/09/2011
• Status: Abandoned Application

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Xie, Xiaohui, Fang, Yikai, Wang, Kongqiao, Rautiainen, Terhi Tuulikki
• Application Number:
  US14/361,439
• Publication Number:
  US 20140324888A1
• Time in Patent Office: Days
• US Class Current: 707/748
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A method, apparatus and computer program product are provided to permit improve gesture recognition based on fusion of different types of sensor signals. In the context of a method, a series of image frames and a sequence of radar signals are received. The method determines an evaluation score for the series of image frames that is indicative of a gesture. This determination of the evaluation score may be based on the motion blocks in an image area and the shift of the motion blocks between image frames. The method also determines an evaluation score for the sequence of radar signals that is indicative of the gesture. This determination of the evaluation score may be based upon the sign distribution in the sequence and the intensity distribution in the sequence. The method weighs each of the evaluation scores and fuses the evaluation scores, following the weighting, to identify the gesture.

Claims

1-22. -22. (canceled)

23. A method comprising:

receiving a series of image frames;

receiving a sequence of radar signals;

determining an evaluation score for the series of image frames that is indicative of a gesture, wherein determining the evaluation score comprises determining the evaluation score based upon motion blocks in an image area and a shift of motion blocks between image frames;

determining an evaluation score for the sequence of radar signals that is indicative of the gesture, wherein determining the evaluation score comprises determining the evaluation score based upon sign distribution in the sequence and intensity distribution in the sequence;

weighting each of the evaluation scores; and

fusing the evaluation scores, following the weighting, to identify the gesture.

24. A method of claim 23 wherein determining the evaluation score for the series of image frames comprises:

down-sampling image data to generated down-sampled image blocks for the series of image frames;

extracting a plurality of features from the down-sampled image blocks; and

determining a moving status of the down-sampled image blocks so as to determine the motion blocks based upon changes in values of respective features in consecutive image frames.

25. A method of claim 24 further comprising determining a direction of motion of the gesture based on movement of a first border and a second border of a projection histogram determined based on the moving status of respective down-sampled image blocks.

26. A method of claim 23 wherein determining an evaluation score for the series of image frames comprises determining the evaluation score based upon a ratio of average motion blocks in the image area.

27. A method of claim 23 wherein a magnitude of the radar signals depend upon a distance between an object that makes the gesture and a radar sensor, and a sign associated with the radar signals depends upon a direction of motion of the object relative to the radar sensor.

28. A method of claim 23 wherein weighting each of the evaluation scores comprises determining weights to be associated with the evaluation scores based upon linear discriminant analysis, Fisher discriminant analysis or linear support vector machine.

29. A method of claim 23 further comprising determining a direction of motion of the gesture based upon the series of image frames in an instance in which the gesture is identified.

30. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the processor, cause the apparatus to:

receive a series of image frames;

receive a sequence of radar signals;

determine an evaluation score for the series of image frames that is indicative of a gesture by determining the evaluation score based upon motion blocks in an image area and a shift of motion blocks between image frames;

determine an evaluation score for the sequence of radar signals that is indicative of the gesture by determining the evaluation score based upon sign distribution in the sequence and intensity distribution in the sequence;

weight each of the evaluation scores; and

fuse the evaluation scores, following the weighting, to identify the gesture.

31. An apparatus of claim 30 wherein the at least one memory and the computer program code are configured to, with the processor, cause the apparatus to determine the evaluation score for the series of image frames by:

down-sampling image data to generated down-sampled image blocks for the series of image frames;

extracting a plurality of features from the down-sampled image blocks; and

determining a moving status of the down-sampled image blocks so as to determine the motion blocks based upon changes in values of respective features in consecutive image frames.

32. An apparatus of claim 31 wherein the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to determine a direction of motion of the gesture based on movement of a first border and a second border of a projection histogram determined based on the moving status of respective down-sampled image blocks.

33. An apparatus of claim 30 wherein the at least one memory and the computer program code are configured to, with the processor, cause the apparatus to determine an evaluation score for the series of image frames by determining the evaluation score based upon a ratio of average motion blocks in the image area.

34. An apparatus of claim 30 wherein a magnitude of the radar signals depend upon a distance between an object that makes the gesture and a radar sensor, and a sign associated with the radar signals depends upon a direction of motion of the object relative to the radar sensor.

35. An apparatus of claim 30 wherein the at least one memory and the computer program code are configured to, with the processor, cause the apparatus to weight each of the evaluation scores by determining weights to be associated with the evaluation scores based upon linear discriminant analysis, Fisher discriminant analysis or linear support vector machine.

36. An apparatus of claim 30 wherein the at least one memory and the computer program code are further configured to, with the processor, determine a direction of motion of the gesture based upon the series of image frames in an instance in which the gesture is identified.

37. The apparatus of claim 30, further comprising user interface circuitry configured to:

facilitate user control of at least some functions of the apparatus through use of a display; and

cause at least a portion of a user interface of the apparatus to be displayed on the display to facilitate user control of at least some functions of the apparatus.

38. A computer program product comprising at least one computer-readable storage medium having computer-executable program code portions stored therein, the computer-executable program code portions comprising program instructions configured to:

receive a series of image frames;

receive a sequence of radar signals;

determine an evaluation score for the series of image frames that is indicative of a gesture by determining the evaluation score based upon motion blocks in an image area and a shift of motion blocks between image frames;

determine an evaluation score for the sequence of radar signals that is indicative of the gesture by determining the evaluation score based upon sign distribution in the sequence and intensity distribution in the sequence;

weight each of the evaluation scores; and

fuse the evaluation scores, following the weighting, to identify the gesture.

39. A computer program product of claim 38 wherein the program instructions configured to determine the evaluation score for the series of image frames comprise program instructions configured to:

down-sample image data to generated down-sampled image blocks for the series of image frames;

extract a plurality of features from the down-sampled image blocks; and

determine a moving status of the down-sampled image blocks so as to determine the motion blocks based upon changes in values of respective features in consecutive image frames.

40. A computer program product of claim 39 wherein the computer-executable program code portions further comprise program instructions configured to determine a direction of motion of the gesture based on movement of a first border and a second border of a projection histogram determined based on the moving status of respective down-sampled image blocks.

41. A computer program product of claim 38 wherein the program instructions configured to determine an evaluation score for the series of image frames comprise program instructions configured to determine the evaluation score based upon a ratio of average motion blocks in the image area.