Method and apparatus for analysing gestures produced in free space, e.g. for commanding apparatus by gesture recognition

• US 20040068409A1
• Filed: 10/05/2003
• Published: 04/08/2004
• Est. Priority Date: 10/07/2002
• Status: Active Grant

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Tanaka, Atau, Fistre, Julien
• Granted Patent: US 7,333,090 B2
• Time in Patent Office: Days
• US Class Current: 704/272
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

The method obtains a signature (S) of a gesture produced in free space, e.g. for producing gesture recognition commands, by deriving at least one time-evolving signal representative of muscular activity involved in producing the gesture and determining at least one value of a parameter yielded by that signal.

The method comprises the steps of:

time dividing the time-evolving signal (14) into sub-frames (SF1, SF2, . . . ), and

for at least one sub-frame:

determining at least one parameter value (Psfi) yielded by the signal over at least a part of that sub-frame, and

expressing said parameter value as a component of a vector (S) along a dimension thereof specifically allocated to that sub-frame (SFi),

the resultant vector yielding the signature (S) of said gesture.

The invention also relates to an apparatus producing such signatures and commands by gesture recognition.

Claims

1. Method of obtaining a signature (S) of a gesture (G) produced in free space, by deriving at least one time-evolving signal (14) representative of muscular activity involved in producing the gesture and determining at least one value of a parameter (Psfi) yielded by that signal,

characterised in that it comprises the steps of:

time dividing the time-evolving signal (14) into sub-frames (SF1, SF2, . . . ), and

for at least one sub-frame (SFi);

determining at least one said parameter value (Psfi) yielded by said time-evolving signal (14) over at least a part of that sub-frame (SFi), and

expressing said parameter value as a component of a vector (S) along a dimension thereof specifically allocated to that sub-frame (SFi),

the resultant vector forming the signature (S) of said gesture (G).

2. Method according to claim 1, wherein said steps of determining a said parameter value (Psfi) and expressing the latter as a component of a vector (S) are performed cyclically on the fly as said time-evolving signal (14) progresses to occupy successive sub-frames (SF1, SF2, . . . ), so that said resultant vector forming the signature (S) acquires an increasing number of dimensions during the progression of said time-evolving signal.

3. Method according to claim 1 or 2, further comprising a gesture identification mode for identifying an analysed gesture among a set of previously learnt gestures (S1-Sy) acquired in a learning mode, each learnt gesture being expressed as a signature vector, wherein a decision on the identification of a gesture under identification is produced while that gesture is still giving rise to an active time-evolving signal (14).

4. Method according to claim 3, wherein each said learnt gesture (S1-Sy) has a determined number of dimensions corresponding to a determined number of sub-frames (SF1-SF8) over which that learnt signature was obtained, wherein said decision on an identification is produced on the basis of fewer sub-frames (SF1-SF4) covered by the time-evolving signal (14) of the gesture under identification, relative to said determined number of sub-frames (SF1-SF8).

5. Method according to any one of claims 1 to 4, further comprising a gesture learning mode comprising, for a given class of gesture (Gk) to be learnt, the steps of:

acquiring a set of signatures (S1-Sy) for that class of gesture by repeatedly producing that gesture and obtaining its signature,

storing said set of signatures, and

storing the correspondence relating said set of signatures to said learned class of gesture (Gk) and, optionally, to a command (Gk) associated to said class of gesture.

6. Method according to claim 5, further comprising a gesture identification mode, in which a gesture under identification (Gc) is currently produced, comprising the steps of:

producing on the fly at least one partial signature vector (Sc) of said gesture under identification, said partial signature vector being limited by the number parameter value(s) (Psfi) currently available, and thereby having fewer dimensions than said learnt signature vectors,

for at least one partial signature vector produced, determining which signature among said learnt signatures best matches said partial signature vector, and

using the result of that determining step to produce a decision on the identification of said gesture under identification.

7. Method according to claim 6, wherein said decision is produced from a number of partial signatures obtained over a period bounded by a fixed limited number of sub-frames (SF1-SF4) which is less than the total number of sub-frames (SF1-SF8) used for obtaining said learnt signatures (S1-Sy).

8. Method according to claim 6, wherein said decision is produced from a number of partial signatures obtained over a period covering a number of sub-frames which is variable as function of the result of said step of determining which signature among said learnt signatures (S1-Sy) best matches said partial signature vector.

9. Method according to any one of claims 1 to 8, further comprising a step of deriving an envelope (16) yielded by said time-evolving signal (14), and wherein said parameter value (Psfi) is a parameter value of said envelope (16).

10. Method according to claim 9, wherein the parameter value (Psfi) is a value of at least one of:

the signal level of the envelope,

the power level of the envelope,

energy level of the envelope,

said value being preferably averaged over the duration (D) of the sub-frame SF1, SF2, . . . ) under consideration.

11. Method according to any one of claims 3 to 10, wherein the duration (D) of each said sub-frame (SF1, SF2, . . . ) in the gesture identification mode is made equal to the duration of its corresponding sub-frame in the learning mode.

12. Method according to any one of claims 1 to 11, wherein said sub-frames (SF1, SF2, . . . ) are substantially uniform in duration (D).

13. Method according to any one of claims 1 to 12, further comprising a step of monitoring the onset of a gesture by detecting the crossing of a threshold (Sthresh) in the amplitude of said time-evolving signal (14), the start of the first sub-frame (SF1) being made to coincide with the time (t0) of said crossing of that threshold.

14. Method according to any one of claims 1 to 13, wherein said time-evolving signal is an electromyographic (EMG) signal from one of a plurality of sensors (2a, 2b), each sensor output being processed as an independent channel.

15. Method according to any one of claim 1 to 12, wherein said time-evolving signal (14) is obtained from at least one sensor (2a, 2b) positioned on a lower limb, preferably a forearm, and more preferably near the top of the forearm, to produce signals representative of a gesture involving a movement of an extremity of said limb, preferably a wrist, hand or fingers.

16. Method according to claim 15, wherein said time-evolving signal (14) is obtained firm sensors (2a, 2b) at antagonistic muscle positions.

17. Method according to any one of claims 1 to 17 applied for commanding an apparatus (12), wherein at least one said signature (Gi) is made to correspond to a specific command (Ci).

18. Method according to claim 17, wherein said gesture is made to correspond to a selected trigger command (Ci) setting the apparatus (12) to enter an operating state, e.g. play, skip, pause etc.

19. Method according to any one of claims 17 to 18, wherein said at least one signature (Si) is derived from a gesture taken from the following set of gestures:

producing a pointing finger,

producing an extended palm,

rotating a wrist,

forming a fist,

deflecting a wrist inwards,

deflecting a wrist outwards,

20. Method according to claim 19, used to command a media player (12) by means of at least one command gesture taken from the following set of command gestures:

producing a pointing finger, e.g. for commanding “

play”

,

producing an extended palm, e.g. for commanding “

stop”

,

rotating a wrist, e.g. for commanding “

fast forward”

, “

rewind”

, or “

song seek/fast play”

,

forming a fist, e.g. for pause/mute,

flexing a wrist (inward), e.g. for commanding “

next track”

or “

step volume up”

;

extending a wrist outwards, e.g. for “

previous track”

or “

step volume down”

21. Method according to any one of claims 1 to 20, further comprising a step of generating a substantially continuously variable command output from a signal (14′

, 16′

) representative of muscular activity involved in producing a gesture.

22. Method according to claim 21, wherein said substantially continuously variable command output is made to vary as a function of the natural oscillations of said signal (14′

) representative of muscular activity involved in producing a gesture, and/or as a function of an envelope (16′

) of that signal (14′

).

23. Method according to claim 21 or 22, wherein said substantially continuously variable command output is independent of a command produced by gesture identification according to any one of claims 17 to 20.

24. Method according to claim 21 or 22, wherein said substantially continuously variable command parameter is made to control a quantity associated to a command produced by gesture identification according to any one of claims 17 to 20.

25. Method according to any one of claims 21 to 24, wherein the signal (14′

) from which said substantially continuously variable command output is extracted and the signal (14) from which said signature is obtained originate from mutually different muscular regions, e.g. from respective arms.

26. Method according to any one of claims 21 to 25 wherein the signal (14′

) from which said substantially continuously variable command output is extracted and the signal (14) from which said signature is obtained originate from common muscular regions, e.g. from the same arm.

28. Apparatus according to claim 27, operative for executing a method according to any one of claims 1 to 26.

32. A computer program product (52) loadable into the memory of a digital computer (48), comprising software code portions for performing the steps of any one of claims 1 to 26 when said product is run on a computer.

27. Apparatus for obtaining a signature (S) of a gesture (G) produced in free space, by deriving at least one time-evolving signal (14) representative of muscular activity involved in producing the gesture and determining at least one value of a parameter (Psfi) yielded by that signal,

characterised in that it comprises:

means (36) for time dividing the time-evolving signal (14) into sub-frames (SF1, SF2, . . . ),

means (34) for determining at least one said parameter value (Psfi) yielded by said time-evolving signal over at least a part of that sub-frame, and

means (42) for expressing said parameter value as a component of a vector (S) along a dimension thereof specifically allocated to that sub-frame (SFi),

the resultant vector forming the signature (S) of said gesture (G).

29. Apparatus according to claim 27 or 28, housed in a unit wearable by a person and further comprising means for sending extracted command inputs to a command executing apparatus (12).

30. A wearable apparatus responsive to user commands, characterised in that it comprises a device according to any one of claims 27 to 29.

31. Use of the apparatus according to any one of claims 27 to 30 for sending user command inputs to an apparatus (12).