Robotic gesture recognition system

• US 7,606,411 B2
• Filed: 10/05/2006
• Issued: 10/20/2009
• Est. Priority Date: 10/05/2006
• Status: Expired due to Fees

• Current Assignee: Department of the Navair Navy
• Original Assignee: the united states of america as represented by the secretary of the navy
• Inventors: Venetsky, Larry, Tieman, Jeffrey W.
• Primary Examiner(s): Le; Brian Q
• Assistant Examiner(s): Woldemariam; Aklilu k
• Application Number:
  US11/586,750
• Publication Number:
  US 20080085048A1
• Time in Patent Office: 1,111 Days
• Field of Search: 382/153, 382/103, 382/104, 382/218, 702/141, 345/475, 348/207.11
• US Class Current: 382/153
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A gesture recognition system enabling control of a robotic device through gesture command by a user is provided, comprising a robotic unit, a video or infrared camera affixed to the robotic unit, computing means, and high and low level of control gesture recognition application code capable of enabling the system to locate points of left hand, right hand, upper torso and lower torso of the user in the video imagery and convert it to waveform data, correlate the waveform data to user command data, and form corresponding control voltage command(s) for production of electric current voltage(s) to drive one or more of the electric motors or actuators of the robotic device to thereby control same. In addition, a computer software program is provided for use in the gesture recognition system described above.

Claims

1. A robotic gesture recognition system to enable control of robotic devices by a human user using gestures can be recognized by the system comprising:

a robotic unit, with one or more electric motors or actuators,

a video camera affixed to the robotic unit, said video camera of recording in pixel, format, panning, tilting and zooming;

a high level control computer in communication with said video camera for enabling recognition of dynamic gestures and static poses of a user in pixel space via video imagery recorded by the video camera;

said high level control computer operating to store and run a high level control gesture recognition application code to transform said dynamic gestures and static poses in pixel space to waveform data, and generating user commands therefrom, said high level control computer including decision-making rules implemented by comparison of said waveform data to a set of thresholds and forces one of the decision rules to be implemented;

a low level control computer in communication with the high level control computer and the robotic unit, said low level control computer storing and running a low level of control gesture recognition application code to transform the user commands received from the high level control computer to control voltage commands;

wherein said robotic unit receives and responds to the control voltage commands received m the low level control computer.

2. The robotic gesture recognition system of claim 1, wherein said robotic unit further comprises panic buttons for manually overriding the gesture recognition system.

3. The robotic gesture recognition system of claim 1, wherein said low level control computer runs computer code for monitoring states of switches on the panic buttons, and deactivating the gesture recognition system upon sensing activation of one or more of said switches.

4. The robotic gesture recognition system of claim 1, further comprising:

a computer display means in communication with the high level control computer; and

a user input means in communication with the high level control computer and the low level control computer.

5. The robotic gesture recognition system of claim 1, wherein said high level control computer enables recognition of dynamic gestures and static poses of a user in pixel space via video imagery recorded by said video camera by means for:

receiving images from said video camera in pixel frames coded in HSL, RGB, or gray scale formats;

locating points of left hand, right hand, upper torso and lower torso of the user in the video imagery, and registering these points in pixel frame coordinates recorded on video image frames;

calculating a relative center point of a relative coordinate system, said relative center point being located at approximately the user'"'"'s cranial region, based on the upper torso point and lower torso point in pixel frame coordinates;

transforming the left hand point and the right hand point in pixel frame coordinates to a left hand point and right hand point in the relative coordinate system;

extracting dynamic and static features of movement of the user from the waveform data for creating data sets for commands for said robotic device; and

decoding dynamic and static features by correlating features to said decision-making rules and generating a user command.

6. The robotic gesture recognition system of claim 5, wherein said high level control computer includes means for:

controlling zoom, pan and tilt camera states; and

comparing the relative left hand point and relative right hand point of a previous video frame to the relative left hand point and relative right hand point of a current video frame, so as to transform the change on relative positions of the left hand points and right hand points to waveform data.

7. The robotic gesture recognition system of claim 1, wherein the low level control computer:

transforms values of motions goals from the high level control computer into desired robotic functions;

determines current states of the electric motors or actuators;

calculates desired states of the electric motors or actuators based on the desired robotic functions; and

computes errors between the current states of the electric motors or actuators and the desired states of the electric motors or actuators;

transforms said computed errors between the current states of the electric motors or actuator and the desired states of the electric motors or actuators into a control voltage command for production of electric current voltage to drive one or more of the electric motors or actuators.

8. The robotic gesture recognition system of claim 1, wherein the low level control computer generates feedback messages for display to the user, to inform the user of lack of or acquisition of control of the robotic device.

9. The robotic gesture recognition system of claim 1, wherein the low level control computer is operable to allow a user to override the gesture recognition application code by inputting direct user command data.

10. The robotic gesture recognition system of claim 1, wherein the video camera is capable of recording in the visible or infrared region.

11. A robotic gesture recognition system to enable control of robotic devices by a human user using gestures can be recognized by the system comprising:

a robotic unit, with one or more electric motors or actuators,

a video camera affixed to the robotic unit, said video camera of recording in pixel, format, panning, tilting and zooming;

a computer including a high level control computer for implementing a high level of control and communicating with said video camera for enabling recognition of dynamic gestures and static poses of a user in pixel space via video imagery recorded by the video camera;

said high level control computer in communication with said video camera and operating to store and a high level of control gesture recognition application code to transform said dynamic gestures and static poses a-user'"'"'s recognized gesture or movement in pixel space to waveform data, and generating user commands therefrom, said high level control computer including decision-making rules implemented by comparison of said waveform data to a set of thresholds and forces one of the decision rules to be implemented;

said computer also including a low level control computer implementing a low level of control and communicating with the robotic unit, said low level control computer storing and running a low level of control gesture recognition application code to transform the user commands received from the high level of control voltage commands;

wherein said robotic unit is in communication with the low level control computer, said robotic unit capable of receiving and responding to the control voltage commands received from e low level control computer.