Method and apparatus for analysing gestures produced in free space, e.g. for commanding apparatus by gesture recognition

• US 7,333,090 B2
• Filed: 10/05/2003
• Issued: 02/19/2008
• Est. Priority Date: 10/07/2002
• Status: Expired due to Fees

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Fistre, Julien, Tanaka, Atau
• Primary Examiner(s): Awad; Amr A.
• Assistant Examiner(s): Sheng; Tom V
• Application Number:
  US10/679,650
• Publication Number:
  US 20040068409A1
• Time in Patent Office: 1,598 Days
• Field of Search: 345/156, 345/158, 715/863, 600/300
• US Class Current: 345/158
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

The method obtains a signature (S) of a gesture produced in free space, e.g. for producing gesture recognition commands, by deriving at least one time-evolving signal representative of muscular activity involved in producing the gesture and determining at least one value of a parameter yielded by that signal.

The method comprises the steps of:

• • time dividing the time-evolving signal (14) into sub-frames (SF1, SF2, . . . ), and
  • for at least one sub-frame:
    • determining at least one parameter value (Psfi) yielded by the signal over at least a part of that sub-frame, and
    • expressing said parameter value as a component of a vector (S) along a dimension thereof specifically allocated to that sub-frame (SFi),
  • the resultant vector yielding the signature (S) of said gesture.

The invention also relates to an apparatus producing such signatures and commands by gesture recognition.

Claims

1. Method of obtaining a signature (S) of a gesture (G) produced in free space, by deriving at least one time-evolving signal (14) representative of muscular activity involved in producing the gesture and determining at least one value of a parameter (Psfi) yielded by that signal, comprising the steps of:

time dividing the time-evolving signal (14) into sub-frames (SF1, SF2, . . . ),

for at least one sub-frame (SFi);

determining at least one said parameter value (Psfi) yielded by said time-evolving signal (14) over at least a part of that sub-frame (SFi), and

expressing said parameter value as a component of a vector (S) along a dimension thereof specifically allocated to that sub-frame (SFi),

the resultant vector forming the signature (S) of said gesture (G); and

identifying an analyzed gesture among a set of previously learnt gestures acquired in a learning mode, each learnt gesture being expressed as a signature vector, wherein a decision on the identification of a gesture under identification is produced while that gesture is still giving rise to an active time-evolving signal (14).

2. Method according to claim 1, wherein said steps of determining a said parameter value (Psfi) and expressing the latter as a component of a vector (S) are performed cyclically on the fly as said time-evolving signal (14) progresses to occupy successive sub-frames (SF1, SF2, . . . ), so that said resultant vector forming the signature (S) acquires an increasing number of dimensions during the progression of said time-evolving signal.

3. Method according to claim 1, wherein each said learnt gesture has a determined number of dimensions corresponding to a determined number of sub-frames (SF1 -SF8) over which that learnt signature was obtained, wherein said decision on an identification is produced on the basis of fewer sub-frames (SF1 -SF4) covered by the time-evolving signal (14) of the gesture under identification, relative to said determined number of sub-frames (SF1-SF8).

4. Method according to claim 1, wherein the gesture learning mode comprises, for a given class of gesture (Gk) to be learnt, the steps of:

acquiring a set of signatures (S1-Sy) for that class of gesture by repeatedly producing that gesture and obtaining its signature,

storing said set of signatures, and

storing the correspondence relating said set of signatures to said learned class of gesture (Gk) and, optionally, to a command associated to said class of gesture.

5. Method according to claim 4, further comprising a gesture identification mode, in which a gesture under identification (Ge) is currently produced, comprising the steps of:

producing on the fly at least one partial signature vector (Sc) of said gesture under identification, said partial signature vector being limited by the number parameter value(s) (Psfi) currently available, and thereby having fewer dimensions than said learnt signature vectors,

for at least one partial signature vector produced, determining which signature among said learnt signatures best matches said partial signature vector, and

using the result of that determining step to produce a decision on the identification of said gesture under identification.

6. Method according to claim 5, wherein said decision is produced from a number of partial signatures obtained over a period bounded by a fixed limited number of sub-frames (SF1 -SF4) which is less than the total number of sub-frames (SF1-SF8) used for obtaining said learnt signatures (S1-Sy).

7. Method according to claim 5, wherein said decision is produced from a number of partial signatures obtained over a period covering a number of sub-frames which is variable as function of the result of said step of determining which signature among said learnt signatures (S1-Sy) best matches said partial signature vector.

8. Method according to claim 1, further comprising a step of deriving an envelope (16) yielded by said time-evolving signal (14), and wherein said parameter value (Psfi) is a parameter value of said envelope (16).

9. Method according to claim 8, wherein the parameter value (Psfi) is a value of at least one of:

the signal level of the envelope,

the power level of the envelope,

energy level of the envelope,

said value being preferably averaged over the duration (D) of the sub-frame (SF1, SF2, . . . ) under consideration.

10. Method according to any one of claim 1, wherein the duration (D) of each said sub-frame (SF 1, SF2, . . . ) in the gesture identification mode is made equal to the duration of its corresponding sub-frame in the learning mode.

11. Method according to claim 1, wherein said sub-frames (SF1, SF2, . . ) are substantially uniform in duration (D).

12. Method according to claim 1, further comprising a step of monitoring the onset of a gesture by detecting the crossing of a threshold (Sthresh) in the amplitude of said time-evolving signal (14), the start of the first sub-frame (SF1) being made to coincide with the time (t0) of said crossing of that threshold.

13. Method according to claim 1, wherein said time-evolving signal is an electromyographic (EMG) signal from one of a plurality of sensors (2a, 2b), each sensor output being processed as an independent channel.

14. Method according to claim 1, wherein said time-evolving signal (14) is obtained from at least one sensor (2a, 2b) positioned on a lower limb, preferably a forearm, and more preferably near the top of the forearm, to produce signals representative of a gesture involving a movement of an extremity of said limb, preferably a wrist, hand or fingers.

15. Method according to claim 14, wherein said time-evolving signal (14) is obtained from sensors (2a, 2b) at antagonistic muscle positions.

16. Method according to claim 1, applied for commanding an apparatus (12), wherein at least one said signature (Si) is made to correspond to a specific command (Ci).

17. Method according to claim 16, wherein said gesture is made to correspond to a selected trigger command (Ci) setting the apparatus (12) to enter an operating state, e.g. play, skip, pause etc.

18. Method according to claim 16, wherein said at least one signature (Si) is derived from a gesture taken from the following set of gestures:

producing a pointing finger,

producing an extended palm,

rotating a wrist,

forming a fist,

deflecting a wrist inwards,

deflecting a wrist outwards.

19. Method according to claim 18, used to command a media player (12) by means of at least one command gesture taken from the following set of command gestures:

producing a pointing finger, e.g. for commanding “

play”

,

producing an extended palm, e.g. for commanding “

stop”

,

rotating a wrist, e.g. for commanding “

fast forward”

, “

rewind”

, or “

song seek/fast play”

,

forming a fist, e.g. for pause/mute,

flexing a wrist (inward), e.g. for commanding “

next track”

or “

step volume up”

;

extending a wrist outwards, e.g. for “

previous track”

or “

step volume down”

.

20. Method according to claim 1, further comprising a step of generating a substantially continuously variable command output from a signal (14′

, 16′

) representative of muscular activity involved in producing a gesture.

21. Method according to claim 20, wherein said substantially continuously variable command output is made to vary as a function of the natural oscillations of said signal (14′

) representative of muscular activity involved in producing a gesture, and/or as a function of an envelope (16′

) of that signal (14′

).

22. Method according to claim 20, wherein said substantially continuously variable command output is independent of a command produced by gesture identification.

23. Method according to claim 20, wherein said substantially continuously variable command parameter is made to control a quantity associated to a command produced by gesture identification.

24. Method according to claim 20, wherein the signal (14′

) from which said substantially continuously variable command output is extracted and the signal (14) from which said signature is obtained originate from mutually different muscular regions, e.g. from respective arms.

25. Method according to claim 20 wherein the signal (14′

) from which said substantially continuously variable command output is extracted and the signal (14) from which said signature is obtained originate from common muscular regions, e.g. from the same arm.

26. A memory with a computer program product (52) stored, which product comprising software code portions for performing the steps of claim 1 when executed by a computer (48).

27. Apparatus for obtaining a signature (S) of a gesture (G) produced in free space, by deriving at least one time-evolving signal (14) representative of muscular activity involved in producing the gesture and determining at least one value of a parameter (Psfi) yielded by that signal, comprising:

means (36) for time dividing the time-evolving signal (14) into sub-frames (SF 1,SF2, . . . ),

means (34) for determining at least one said parameter value (Psfi) yielded by said time-evolving signal over at least a part of that sub-frame,

means (42) for expressing said parameter value as a component of a vector (S) along a dimension thereof specifically allocated to that sub-frame (SFi), the resultant vector forming the signature (S) of said gesture (G), and

means for identifying an analyzed gesture among a set of previously learnt gestures acquired in a learning mode, each learnt gesture being expressed as a signature vector, wherein a decision on the identification of a gesture under identification is produced while that gesture is still giving rise to an active time-evolving signal (14).

28. Apparatus according to claim 27, housed in a unit wearable by a person and further comprising means for sending extracted command inputs to a command executing apparatus (12).

29. A wearable apparatus responsive to user commands, comprising a device according to claim 27.

30. Use of the apparatus according to claim 27 for sending user command inputs to an apparatus (12).