Controlling motion using a human machine interface

• US 7,802,193 B1
• Filed: 06/05/2007
• Issued: 09/21/2010
• Est. Priority Date: 12/19/2001
• Status: Active Grant

• Current Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Original Assignee: Sandia Corporation (Martin Marietta Corporation)
• Inventors: Anderson, Robert J., McDonald, Michael J., Gladwell, T. Scott
• Primary Examiner(s): Pillai; Namitha
• Application Number:
  US11/758,139
• Time in Patent Office: 1,204 Days
• Field of Search: 715/757, 715/848, 715/751, 715/811
• US Class Current: 715/757
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/35506 Camera images overlayed with graphics, model
• CPC Class Codes: G05B 2219/37567 3-D vision, stereo vision, with two cameras
• CPC Class Codes: G05B 2219/40161 Visual display of machining, operation, remote viewing
• CPC Class Codes: G05B 2219/40169 Display of actual situation at the remote site
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics

Abstract

Machine motion is directed using a graphical interface by establishing an icon in an image, displaying the icon in images having perspectives such that the icon is rendered in a corresponding relative position in the second images and moving the icons in the second images when the icon is moved in the first image, the movement being constrained along a line projected from a camera point associated with the first image and a target destination. Actions are defined using the icon and a set of action descriptors. The dominant motion space is constrained to a spherical coordinate frame centered on a vision locus and machine motions are mapped to coincide therewith based on a graphical interface view.

Claims

1. A Human Machine Interface (HMI) for directing motion of an element relative to an object in a 3D environment using a graphical interface, the HMI comprising:

a display operatively configured to display the object in a first image observed from a first viewpoint and a second image observed from a second viewpoint, the first and second viewpoints having know coordinates within a coordinate system of the 3D environment;

an input device operatively configured to;

select a geometric entity based upon a desired interaction of the element with the object, the geometric entity selected from the group consisting of a point, a line and a plane; and

,

establish a number of icons at locations in the first image, the number of the icons determined by the geometric entity selected, the icons delineating the geometric entity as viewed in the first image, the locations of the icons in the first image corresponding to the object as viewed in the first image; and

,

a processor coupled to the display and the input device, the processor operatively configured to;

calculate for the second image, projections of the locations of the icons in the first image, the projection of each location of an icon in the first image comprising an epipolar line in the second image;

generate the first image having the icons placed at their respective locations in the first image and, generate the second image having the icons placed at their respective locations in the second image thereby delineating the geometric entity in the second image; and

,

determine a constraint, whereby the respective locations of icons in the second image are constrained to exist along their respective epipolar lines and, wherein movement of an icon along its corresponding epipolar line in the second image, does not change the location of the icon as viewed in the first image.

2. The Human Machine Interface (HMI) of claim 1 wherein the processor is further configured to:

apply an action command to the geometric entity in the second image; and

,

generate control commands including a trajectory and a velocity to direct the element in the 3D environment toward a primary goal.

3. The Human Machine Interface (HMI) of claim 2 wherein the processor is further configured to:

adjust the motion of the element if a satisfactory goal is reached before the primary goal by using an input device with one of at least two linear and three digital degrees of freedom.

4. The Human Machine Interface (HMI) of claim 3 wherein the input device is at least one of three buttons, a lever-dash pot, a mouse and a joystick.

5. The Human Machine Interface (HMI) of claim 4 wherein adjusting the motion of the element includes the ability to move along the trajectory with a negative velocity, thereby enabling element back up control.

6. The Human Machine Interface (HMI) of claim 1 wherein the processor is further configured to:

describe one or more tasks as one or more geometric entities overlaying an image; and

,

associate one or more commands with the one or more icons to complete the one or more tasks.

7. The Human Machine Interface (HMI) of claim 6 further comprising:

a storage device for storing the one or more tasks for retrieval and modification of the one or more tasks.

8. The Human Machine Interface (HMI) of claim 6 further comprising:

a VCR type control for controlling the one or more tasks.

9. The Human Machine Interface (HMI) of claim 8 wherein the VCR type control comprises:

a forward button for stepping through a task in a forward sequence;

a reverse button for stepping through a task in a backward sequence; and

,

a pause button for stopping a pending task operation.

10. The Human Machine Interface (HMI) of claim 9 wherein the VCR type control further comprises:

at least one auxiliary button directed to specific functionalities of a device being controlled.

11. The Human Machine Interface (HMI) of claim 8 wherein the VCR type control is activated when the one or more tasks is retrieved.

12. The Human Machine Interface (HMI) of claim 1 further comprising first and second image acquisition devices to acquire the first and second images respectively.

13. The Human Machine Interface (HMI) of claim 1 wherein an icon represents at least one of a component of a robot, a machine that can move in at least three axes, and a virtual object that is movable in at least three axes in the 3D environment.

14. The Human Machine Interface (HMI) of claim 1 wherein the processor is further configured to:

establish additional icons that are used in combination with the number of icons to define at least one of directed motion, alignment lines, and motion limit points.

15. The Human Machine Interface (HMI) of claim 1 wherein the processor is further configured to:

selectively transition between automatic motion control, telerobotic control, and teleoperation, based upon user input.

16. The Human Machine Interface (HMI) of claim 15 wherein telerobotic control moves the element in real time corresponding to movement of an icon in the second image and wherein element motion is controlled as trajectories defined by the movement of the icon in the second image with user-controlled velocity.

17. The Human Machine Interface of claim 1 wherein the processor is further configured to display one or more epipolar lines in the second image.