Mobile camera-space manipulation

• US 6,194,860 B1
• Filed: 11/01/1999
• Issued: 02/27/2001
• Est. Priority Date: 11/01/1999
• Status: Expired due to Fees

• Current Assignee: Yoder Software Incorporated
• Original Assignee: Yoder Software Incorporated
• Inventors: Seelinger, Michael J., Skaar, Steven B., Yoder, John-David S.
• Primary Examiner(s): Ip, Paul
• Application Number:
  US09/430,940
• Time in Patent Office: 484 Days
• Field of Search: 318/560-696, 364/424.02, 364/424.06, 395/89, 395/94, 901/1, 901/7, 901/47, 901/50, 901/46, 33/288, 33/203.18, 356/155, 356/375, 701/25, 701/50, 701/28, 187/224, 187/231, 180/169, 414/274, 414/730, 358/103
• US Class Current: 318/587
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B66F 9/0755 Position control; Position detectors
• CPC Class Codes: G05B 2219/39391 Visual servoing, track end effector with camera image feedback
• CPC Class Codes: G05B 2219/40513 Planning of vehicle and of its manipulator arm
• CPC Class Codes: G05B 2219/40564 Recognize shape, contour of object, extract position and orientation
• CPC Class Codes: G05B 2219/40604 Two camera, global vision camera, end effector neighbourhood vision camera
• CPC Class Codes: G05B 2219/40611 Camera to monitor endpoint, end effector position
• CPC Class Codes: G05B 2219/45049 Forklift

Abstract

The invention is a method of using computer vision to control systems consisting of a combination of holonomic and nonholonomic degrees of freedom such as a wheeled rover equipped with a robotic arm, a forklift, and earth-moving equipment such as a backhoe or a front-loader. Using vision sensors mounted on the mobile system and the manipulator, the system establishes a relationship between the internal joint configuration of the holonomic degrees of freedom of the manipulator and the appearance of features on the manipulator in the reference frames of the vision sensors. Then, the system, perhaps with the assistance of an operator, identifies the locations of the target object in the reference frames of the vision sensors. Using this target information, along with the relationship described above, the system determines a suitable trajectory for the nonholonomic degrees of freedom of the base to follow towards the target object. The system also determines a suitable pose or series of poses for the holonomic degrees of freedom of the manipulator. With additional visual samples, the system automatically updates the trajectory and final pose of the manipulator so as to allow for greater precision in the overall final position of the system.

Claims

1. A mobile camera space manipulation means comprising:

a multiple degree of freedom movement manipulator means including;

a base means,

an end effector means,

connection means between the base means and the end effector means allowing multiple degrees of freedom of movement of the end effector means with respect to the base means based on known nominal kinematics, and including first sensing means for producing signals correlated to the orientation of the connection means relative to the base means,

motive means associated with the base means for allowing locomotion of the base means in a plurality of degrees of freedom of movement and including second sensing means for producing signals correlated to the distance and direction of movement of the base means along a surface;

first cue means associated with the end effector means;

second cue means associated with a work object;

two or more camera means each having a field of view being positionable to capture at least intermittently both the first and second cue means in a field of view, each camera means being attached to the base means;

camera space means associated with each camera means to convert the field of view of each camera means into a corresponding two dimensional camera space; and

processing means including;

distinguishing means to distinguish the first and second cue means from generally all other contents of the camera spaces;

tracking means to obtain and store information relating to position and movement of the cue means in the camera spaces, monitoring means to obtain and store information relating to holonomic orientation and position of the connection means relative to the base means from the first sensing means, and nonholonomic history of movement of the base means from the second sensing means;

estimation and planning means to repetitively propose a plan of movement for one or both of the connection means and motive means to bring about a desired positional relationship between the first and second cue means in the camera spaces, the plan being based on information in the tracking means and the monitoring means taking into consideration both holonomic and nonholonomic relationships;

control means for instructing movements of one or both of the connection and motive means to follow the plan in physical space.

2. The mobile camera space manipulation means of claim 1 wherein the base means includes a plurality of wheels for allowing mobility over a surface.

3. The mobile camera space manipulation means of claim 1 wherein the base means consists of a mobile cart.

4. The mobile camera space manipulation means of claim 1 wherein the base means consists of an underwater vehicle.

5. The mobile camera space manipulation means of claim 1 wherein the end effector consists of a grasping means.

6. The mobile camera space manipulation means of claim 1 wherein the end effector consists of a fork lift.

7. The mobile camera space manipulation means of claim 1 wherein the base means includes a plurality of wheels for allowing mobility over a surface.

8. The mobile camera space manipulation means of claim 1 wherein the base means consists of a mobile cart.

9. The mobile camera space manipulation means of claim 1 wherein the base means consists of an underwater vehicle.

10. The mobile camera space manipulation means of claim 1 wherein the end effector consists of a grasping means.

11. The mobile camera space manipulation means of claim 1 wherein the end effector consists of a fork lift.

13. The mobile camera space manipulation means of claim 1 wherein the base means includes a plurality of wheels for allowing mobility over a surface.

14. The mobile camera space manipulation means of claim 1 wherein the base means consists of a mobile cart.

15. The mobile camera space manipulation means of claim 1 wherein the base means consists of an underwater vehicle.

16. The mobile camera space manipulation means of claim 1 wherein the end effector consists of a grasping means.

17. The mobile camera space manipulation means of claim 1 wherein the end effector consists of a fork lift.

12. A camera space manipulation control means, utilizing two or more camera means for engaging an end effector means with a work object, comprising:

an articulateable manipulator means of known nominal kinematics in physical space extending from a mobile base to an outward end for movement of the outward end in a predefined physical work space in the physical space relative to the mobile base, the mobile base having a nonholonomic kinematic relationship between wheel rotation and base-position response, the manipulator means including a motor means for articulating the manipulator means in said physical space, the mobile base having a motor means and a steering means to locate the mobile base in any direction over a surface or in three dimensions, and means for producing a signal identifying an approximate position and orientation of the manipulator means with respect only to the base, wherein the kinematic description of the manipulator means with base being known and the kinematic description of the mobile base being known only relative to prior movement;

each camera means being positioned on the mobile base and each camera means being oriented generally towards the end effector means for providing camera vision intermittently of the end effector means and the work object in camera space;

first visual cue means associated with the end effector means;

second visual cue means associated with the work object, the first and second visual cue means comprising means which are distinct and identifiable in said camera space manipulation control means in any surrounding environment, the first and second visual cue means providing descriptions of three dimensional physical space maneuver objectives as admissible configurations of visual cue means in the two dimensional camera spaces of the camera means; and

a control means operatively connected to the manipulator means and the camera means, the control means including computing means for receiving the signal from the manipulator means and identifying the approximate position and orientation of the manipulator means with respect to the base means through the use of previously known kinematics, and signal processing means which identifies and tracks the visual cue means in the camera spaces to convert such into two dimensional camera space cue position signals, the manipulator approximate position and orientation signal and the camera space cue position signals being used in the control means to estimate the relationship between the position and orientation of the manipulator means and the location in each camera space of the visual cue means placed on the manipulator means, and using the current estimations of these relationships selecting required movement and orientation of the manipulator means which will bring about admissible configurations of the visual cue means in each camera space to insure successful engagement of the object in physical space, and to control orientation commands resulting from the estimated relationship.

18. A method of camera space manipulation utilizing at least two camera means for engaging an articulateable manipulator means with an object where there is not any known prior three dimensional physical space relationship between the manipulator means and the object, and there is a known three dimensional physical space relationship between the manipulator means and physical space in a two dimensional image at the focal plane of the camera means, denoted as camera space, comprising the steps:

orienting each camera means to view the manipulator means which has an arm extending from a base to an outward end which is moveable in physical work space with known nominal kinematics relative to the base;

the manipulator means including a motor means which articulates the manipulator means in said physical work space, and means for producing a signal identifying the approximate position and orientation of the manipulator means with respect only to the base in said physical work space;

the base having motor and steering means for moving the base in any direction along a surface and including means for producing a signal identifying the approximate position and orientation of the base, each camera means being positioned and oriented to provide, at least intermittently, camera vision of at least the outward end of the manipulator means in at least part of the physical work space to view at least the outer end of the manipulator means and the work object in camera space;

placing a first visual cue means in association with an outward end of the arm;

placing a second visual cue means in association with the object to be engaged by the manipulator means, the first and second visual cue means comprising means which are distinct and identifiable in said camera space from the remainder of the system and any surrounding environment, the first and second visual cue means providing descriptions of three dimensional physical space maneuver objectives in terms of admissible configurations of the visual cue means in the two dimensional camera space of each camera;

receiving signals from the manipulator means and base means and identifying the approximate position and orientation of the manipulator means and base means with respect to the base and surface respectively through the use of known nominal kinematics;

identifying and tracking the visual cue means in the two dimensional camera space of each camera means and respectively estimating the relationship between the position and orientation of the manipulator means and the location in each camera space of the visual cue means placed on the manipulator means, and using the current estimation of these relationships to select the movement and to command the orientation of the manipulator means which will bring about the admissible configurations of the visual cue means in each camera space which insures successful engagement of the object; and

continuously controlling movement and orientation of the manipulator means according to such autonomously selected movement and orientation commands to achieve engagement of the manipulator means with the work object in said physical work space.