Robotic gesture recognition system

• US 20080085048A1
• Filed: 10/05/2006
• Published: 04/10/2008
• Est. Priority Date: 10/05/2006
• Status: Active Grant

• Current Assignee: Department of the Navair Navy
• Original Assignee: US Department of The Navy (U.S. Department Of Defense)
• Inventors: Venetsky, Larry, Tieman, Jeffrey W.
• Granted Patent: US 7,606,411 B2
• Time in Patent Office: Days
• US Class Current: 382/153
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A gesture recognition system enabling control of a robotic device through gesture command by a user is provided, comprising a robotic unit, a video or infrared camera affixed to the robotic unit, computing means, and high and low level of control gesture recognition application code capable of enabling the system to locate points of left hand, right hand, upper torso and lower torso of the user in the video imagery and convert it to waveform data, correlate the waveform data to user command data, and form corresponding control voltage command(s) for production of electric current voltage(s) to drive one or more of the electric motors or actuators of the robotic device to thereby control same. In addition, a computer software program is provided for use in the gesture recognition system described above.

Claims

1. A robotic gesture recognition system to enable control of robotic devices by a human user using gestures and movements that can be recognized by the system comprising:

a robotic unit, with one or more electric motors or actuators, motor control amplifiers, power source, omni-directional wheels, gear boxes, vehicle body, chassis with suspension, electric wiring and cabling, control and power supply circuitry;

a video camera affixed to the robotic unit, said video camera capable of recording in pixel format, panning, tilting and zooming;

a high level control computer in communication with said video camera for enabling recognition of dynamic gesture signals and static poses of a user in pixel space via video imagery recorded by the video camera;

said high level control computer operating to store and run a high level of control gesture recognition application code to transform a user'"'"'s recognized gesture or movement in pixel space to waveform data, and generating user commands therefrom; and

a low level control computer in communication with the high level control computer and the robotic unit, said low level control computer capable of storing and running a low level of control gesture recognition application code to transform the user commands received from the high level of control computer to control voltage commands;

wherein said robotic unit receives and responds to the control voltage commands received from the low level control computer.

2. The robotic gesture recognition system of claim 1, wherein said robotic unit further comprises panic buttons for manually overriding the gesture recognition system.

3. The robotic gesture recognition system of claim 1, wherein said low level control computer runs computer code capable of monitoring states of switches on the panic buttons, and deactivating the gesture recognition system upon sensing activation of one or more of said switches.

4. The robotic gesture recognition system of claim 1, further comprising:

a computer display means in communication with the high level control computer; and

a user input means in communication with the high level control computer and the low level control computer.

5. The robotic gesture recognition system of claim 1, wherein said high level control computer enables recognition of dynamic gesture signals and static poses of a user in pixel space via video or infrared imagery recorded by said video camera as an input means by means of application software capable of:

receiving images from said video camera in pixel frames coded in HSL, RGB, or gray scale formats;

locating points of left hand, right hand, upper torso and lower torso of the user in the video imagery, and registering these points in pixel frame coordinates recorded on the video image frames;

calculating a relative center point of a relative coordinate system, said relative center point being located at approximately the user'"'"'s cranial region, based on the upper torso point and lower torso point in pixel frame coordinates;

transforming the left hand point and the right hand point in pixel frame coordinates to a left hand point and right hand point in the relative coordinate system;

extracting dynamic and static features of movement of the user from the waveform data for creating data sets for commands for said robotic device; and

decoding dynamic and static features by correlating features to user command rules and generating a user command.

6. The robotic gesture recognition system of claim 5, wherein said application software also includes the features of:

controlling zoom, pan and tilt camera states; and

comparing the relative left hand point and relative right hand point of a previous video frame to the relative left hand point and relative right hand point of a current video frame, so as to transform the change on relative positions of the left hand points and right hand points to waveform data.

7. The robotic gesture recognition system of claim 1, wherein the low level control computer:

transforms values of motions goals from the high level control computer into desired robotic functions;

determines current states of electric motors or actuators;

calculates desired states of the electric motors or actuators based on the desired robotic functions; and

computes errors between the current states of the electric motors or actuators and the desired states of the electric motors or actuators;

transforms said computed errors between the current states of the electric motors or actuator and the desired states of the electric motors or actuators into a control voltage command for production of electric current voltage to drive one or more of the electric motors or actuators.

8. The robotic gesture recognition system of claim 1, wherein the low level control computer generates feedback messages for display to the user, to inform the user of lack of or acquisition of control of the robotic device.

9. The robotic gesture recognition system of claim 1, wherein the low level control computer is operable to allow a user to override the computer application software by inputting direct user command data.

10. The robotic gesture recognition system of claim 1, wherein the video camera is capable of recording in the visible or infrared region.

11. A robotic gesture recognition system to enable control of robotic devices by a human user using gestures and movements that can be recognized by the system comprising:

a robotic unit, with one or more electric motors or actuators, motor control amplifiers, power source, omni-directional wheels, gear boxes, vehicle body, chassis with suspension, electric wiring and cabling, control and power supply circuitry;

a video camera affixed to the robotic unit, said video camera capable of recording in pixel format, panning, tilting and zooming; and

a computer including a high level control computer for implementing a high level of control and communicating with said video camera for enabling recognition of dynamic gesture signals and static poses of a user in pixel space via video imagery recorded by the video camera;

said high level control computer in communication with said video camera and operating to store and run a high level of control gesture recognition application code to transform a user'"'"'s recognized gesture or movement in pixel space to waveform data, and generating user commands therefrom;

said computer also including a low level control computer implementing a low level of control and communicating with the robotic unit, said low level control computer capable of storing and running a low level of control gesture recognition application code to transform the user commands received from the high level of control voltage commands;

wherein said robotic unit is in communication with the low level control computer, said robotic unit capable of receiving and responding to the control voltage commands received from the low level control computer.

12. A method of identifying a human gesture, comprising:

providing a time sequence of pixel images of the human gesture;

transforming the time sequence of pixel images into waveforms;

extracting features from the waveforms; and

identifying the human gesture based on the extracted features.

13. The method of claim 12 wherein the providing step includes providing a time sequence of pixel images of the human gesture using a video camera.

14. The method of claim 12 wherein the providing step includes providing a time sequence of pixel images of the human gesture using an infrared camera.

15. The method of claim 12 wherein the human gesture is an aircraft taxiing gesture.

16. The method of claim 12 wherein the extracting step includes extracting static features from the waveforms.

17. The method of claim 12 wherein the extracting step includes extracting dynamic features from the waveforms.

18. The method of claim 12 wherein the transforming step includes transforming the time sequence of pixel images into a right hand waveform and a left hand waveform.

19. The method of claim 18 wherein the extracting step includes extracting hand position, amplitude, phase and frequency from the right hand waveform and the left hand waveform.

20. The method of claim 12 wherein the transforming step includes transforming the time sequence of pixel images into waveforms that are normalized to a head location.

21. The method of claim 12 wherein the step of providing includes providing a time sequence of pixel images of the human gesture using an error filtering method.

22. The method of claim 21 wherein the error filtering method is one of a data delay method, a statistical method and a least squares estimate method.

23. The method of claim 19 wherein the identifying step includes identifying the human gesture by comparing one or more of the hand position, amplitude, phase and frequency from the right hand waveform and the left hand waveform to at least one rule that describes the human gesture.

24. The method of claim 23 wherein the human gesture is one of forward, back, left, right and stop.

25. The method of claim 23 wherein the identifying step includes identifying the human gesture using a Fuzzy Adaptive Resonance Theory Neural Network.

26. A computer readable medium containing a computer program for performing a methoc of identifying a human gesture, the method comprising transforming a time sequence of pixel images into waveforms;

extracting features from the waveforms; and

identifying the human gesture based on the extracted features.

27. An apparatus for identifying a human gesture, comprising:

means for providing a time sequence of pixel images of the human gesture; and

a computer connected to the means for providing, the computer including means for transforming the time sequence of pixel images into waveforms, means for extracting features from the waveforms and means for identifying the human gesture based on the extracted features.

28. The apparatus of claim 27 wherein the means for providing comprises one of a video camera and an infrared camera.

29. The apparatus of claim 27 wherein the human gesture is an aircraft taxiing gesture.

30. The apparatus of claim 27 wherein the means for transforming includes means for transforming the time sequence of pixel images into a right hand waveform and a left hand waveform.

31. The apparatus of claim 27 wherein the means for extracting includes means for extracting hand position, amplitude, phase and frequency from the right hand waveform and the left hand waveform.

32. The apparatus of claim 27 wherein the means for transforming includes means for transforming the time sequence of pixel images into waveforms that are normalized to a head location.

33. The apparatus of claim 27 wherein the means for providing includes means for providing a time sequence of pixel images of the human gesture using an error filtering method.

34. The apparatus of claim 27 wherein the means for identifying includes means for identifying the human gesture by comparing one or more of the hand position, amplitude, phase and frequency from the right hand waveform and the left hand waveform to at least one rule that describes the human gesture.

35. The apparatus of claim 34 wherein the means for identifying includes means for identifying the human gesture using a Fuzzy Adaptive Resonance Theory Neural Network.