Manipulation control based upon mimic of human gestures

• US 9,052,710 B1
• Filed: 08/26/2009
• Issued: 06/09/2015
• Est. Priority Date: 03/20/2009
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Exelis Incorporated (L3Harris Technologies, Inc.)
• Inventors: Farwell, Mark L.
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US12/547,793
• Time in Patent Office: 2,113 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: G05B 19/423 Teaching successive positions by walk-through, i.e. the tool head or end effector being grasped and guided directly, with or without servo-assistance, to follow a path
• CPC Class Codes: G05B 19/427 Teaching successive positions by tracking the position of a joystick or handle to control the positioning servo of the tool head, master-slave control G05B19/423 takes precedence
• CPC Class Codes: G05B 2219/35448 Datasuit, arm sleeve, actor, operator wears datasuit and generates motion
• CPC Class Codes: G05B 2219/36442 Automatically teaching, teach by showing

Abstract

A human gesture tracking system includes a sensor device with at least one sensor unit, where the sensor unit is configured to measure at least one of position in three dimensions of a portion of an operator'"'"'s limb and orientation in three dimensions of the portion of the operator'"'"'s limb, a processor configured to receive information from the sensor device so as to determine orientation and position displacement in six degrees of freedom of the operator'"'"'s limb, and an activator that is coupled to communicate with the processor and is further actuated by the operator to designate a start time and an end time in which position and orientation displacements of the portion of the operator'"'"'s limb are determined by the processor. During system operation, the processor determines position and orientation displacements of the portion of the operator'"'"'s limb in tracking sessions, each tracking session is defined as a time period between the start time and the end time as designated by the activator. The processor further determines an initial position and orientation of the portion of the operator'"'"'s limb at the start time of each tracking session and determines position and orientation displacements of the portion of the operator'"'"'s limb during each tracking session in relation to the initial position and orientation for the tracking session.

Claims

1. A human gesture tracking system comprising:

a sensor device comprising at least one sensor unit configured to couple with an operator'"'"'s limb, wherein each sensor unit is configured to measure at least one of position in three dimensions of a portion of the operator'"'"'s limb and orientation in three dimensions of the portion of the operator'"'"'s limb;

a processor configured to receive information from the sensor device so as to determine orientation and position displacement in six degrees of freedom of the operator'"'"'s limb; and

an activator that is coupled to communicate with the processor and is further actuated by the operator to designate a start time and an end time in which position and orientation displacements of the portion of the operator'"'"'s limb are determined by the processor;

wherein;

during system operation, the processor determines position and orientation displacements of the portion of the operator'"'"'s limb in tracking sessions, each tracking session is defined as a time period between the start time and the end time as designated by the activator, the processor determines an initial position and orientation of the portion of the operator'"'"'s limb at the start time of each tracking session and determines position and orientation displacements of the portion of the operator'"'"'s limb during each tracking session in relation to the initial position and orientation for the tracking session; and

the system is configured such that each tracking session effects an occurrence of a translated movement within a second system prior to establishing a subsequent tracking session such that a gesture by the operator that is defined by movement of the operator'"'"'s limb during each tracking session effects a corresponding mimic movement by a component of the second system.

2. The system of claim 1, wherein the portion of the operator'"'"'s limb comprises the operator'"'"'s hand, and the system further comprises:

a rod configured to be held by the operator'"'"'s hand, wherein the rod includes the sensor device and the activator.

3. The system of claim 2, wherein the activator comprises a switch that is depressed by the operator during system operation to designate at least one of the start time and the end time.

4. The system of claim 2, wherein the sensor device comprises a single sensor unit that measures position information in three dimensions and orientation information in three dimensions of the operator'"'"'s hand based upon manipulation of the rod by the operator'"'"'s hand.

5. The system of claim 1, wherein the sensor device comprises a plurality of sensor units, and the processor utilizes orientation information in three dimensions measured by the sensor units to determine orientation and position displacement in six degrees of freedom of the operator'"'"'s limb.

6. The system of claim 5, wherein the portion of the operator'"'"'s limb comprises the operator'"'"'s hand, and the sensor device includes a first sensor unit that is secured between a shoulder joint and an elbow joint of the operator'"'"'s limb, a second sensor unit that is secured between the elbow joint and a wrist joint of the operator'"'"'s limb, and a third sensor unit that is secured proximate the operator'"'"'s hand.

7. The system of claim 6, further comprising:

a rod configured to be held by the operator'"'"'s hand, wherein the rod includes the third sensor unit and the activator.

8. The system of claim 1, wherein the component of the second system comprises:

a robotic manipulator device including an end-effector that is controlled by the robotic manipulator device to be positioned in a plurality of positions and orientations in six degrees of freedom with respect to a reference location of the robotic manipulator device;

wherein the processor provides orientation and position displacement information of each tracking session to the robotic manipulator device to facilitate a corresponding position and orientation displacement of the end-effector with respect to the reference location of the robotic manipulator device such that the end-effector mimics a movement in real time of the portion of the operator'"'"'s limb during each tracking session.

9. The system of claim 8, wherein the robotic manipulator includes a plurality of rigid sections and a plurality of joints connected with and about which corresponding rigid sections rotate to achieve a selected position and orientation alignment of the end-effector with respect to the reference location of the robotic manipulator device, the at least one sensor unit of the sensor device is secured to at least one rigid support member, and a number of rigid sections of the robotic manipulator differs from a number of rigid support members to which the at least one sensor unit is secured.

10. The system of claim 1, wherein the system is configured to connect as a peripheral to a computer system to facilitate transmission of orientation and position displacement information relating to movement of the operator'"'"'s limb from the processor to the computer system so as to control operation of a software program operated by the computer system.

22. The system of claim 1, wherein position and orientation displacements of the portion of the operator'"'"'s limb are determined only between the start time and the end time of the tracking sessions such that each tracking session is calibrated separately and independently from all other tracking sessions.

11. A method of tracking and mimicking human gestures using a human gesture tracking system, the method comprising:

(a) actuating an activator of the system to designate a start time of a tracking session;

(b) measuring, via a sensor device of the system that is coupled with an operator'"'"'s limb, initial information comprising at least one of position in three dimensions of a portion of the operator'"'"'s limb and orientation in three dimensions of the portion of the operator'"'"'s limb at the start time of the tracking session;

(c) providing the measured information of step (b) from the sensor device to the processor of the system;

(d) determining, via the processor of the system, an initial position and orientation of a portion of the operator'"'"'s limb at the start time of the tracking session from the measured information provided in step (c);

(e) measuring, via the sensor device, current information of at least one of position in three dimensions of the portion of the operator'"'"'s limb and orientation in three dimensions of the portion of the operator'"'"'s limb during the tracking session;

(f) providing the measured information of step (e) from the sensor device to the processor of the system;

(g) determining, via the processor of the system, a current position and orientation of the portion of the operator'"'"'s limb from the measured information provided in step (f);

(h) determining, via the processor of the system, position and orientation displacement information of the portion of the operator'"'"'s limb based upon a change from initial position and orientation of the operator'"'"'s limb as determined in step (d) to a current position and orientation of the operator'"'"'s limb as determined in step (g);

(i) providing the position and orientation displacement information determined in step (h) to a second system that facilitates control of the second system based upon the position and orientation displacement information; and

(j) repeating steps (e)-(i) until the activator of the system is actuated to designate an end time of the tracking session;

wherein each tracking session effects an occurrence of a translated movement within the second system prior to establishing a subsequent tracking session such that a gesture by the operator that is defined by movement of the operator'"'"'s limb during each tracking session effects a corresponding mimic movement by a component of the second system.

12. The method of claim 11, further comprising:

(k) actuating the activator of the system to designate a start time of a second tracking session;

(l) measuring, via a sensor device of the system, initial information comprising at least one of position in three dimensions of a portion of an operator'"'"'s limb and orientation in three dimensions of the portion of the operator'"'"'s limb at the start time of the second tracking session;

(m) providing the measured information of step (l) from the sensor device to the processor of the system;

(n) determining, via the processor of the system, an initial position and orientation of a portion of the operator'"'"'s limb at the start time of the second tracking session from the measured information provided in step (m);

(o) repeating steps (e)-(i) until the activator of the system is actuated to designate an end time of the second tracking session.

13. The method of claim 11, wherein a plurality of tracking sessions are generated according to steps (a)-(j), and initial position and orientation information is determined at the start time of each tracking session.

14. The method of claim 11, wherein the portion of the operator'"'"'s limb comprises the operator'"'"'s hand, the sensor device comprises a sensor unit connected with a rod that is held by the operator'"'"'s hand.

15. The method of claim 14, wherein the activator comprises a switch connected with the rod, and the actuation of the switch in at least one of steps (a) and (j) comprises depressing the switch.

16. The method of claim 14, wherein the sensor device comprises a single sensor unit that, in steps (b) and (e), measures position information in three dimensions and orientation information in three dimensions of the operator'"'"'s hand based upon manipulation of the rod by the operator'"'"'s hand.

17. The method of claim 11, wherein the sensor device comprises a plurality of sensor units that, in steps (b) and (e), measure orientation information in three dimensions of the operator'"'"'s limb, and the processor, in steps (d) and (g), determines position and orientation of the portion of the operator'"'"'s limb based upon the orientation information provided by the sensor units.

18. The method of claim 17, wherein the portion of the operator'"'"'s limb comprises the operator'"'"'s hand, and the sensor device includes a first sensor unit that is secured between a shoulder joint and an elbow joint of the operator'"'"'s limb, a second sensor unit that is secured between the elbow joint and a wrist joint of the operator'"'"'s limb, and a third sensor unit that is secured proximate the operator'"'"'s hand.

19. The method of claim 18, wherein the third sensor unit is secured to a rod configured that is held by the operator'"'"'s hand.

20. The method of claim 11, wherein the second component of the second system comprises a robotic manipulator device including an end-effector, and step (i) further comprises:

(i.1) controlling movement of the end-effector, via the position and orientation displacement information provided to the robotic manipulator device such that the end-effector moves in relation to a reference location of the robotic manipulator device such that the end-effector mimics a corresponding movement in real time of the operator'"'"'s limb.

21. The method of claim 11, wherein the second system comprises a computer system, and the human gesture tracking system couples as a peripheral device for the computer system, and step (i) further comprises:

(i.1) controlling operation of a software program operated by the computer system based upon the position and orientation displacement information provided to the computer system.