Nonholonomic camera space manipulation

• US 5,300,869 A
• Filed: 07/01/1993
• Issued: 04/05/1994
• Est. Priority Date: 07/30/1992
• Status: Expired due to Term

• Current Assignee: Iowa State University Research Foundation Incorporated (Iowa State University)
• Original Assignee: Iowa State University Research Foundation Incorporated (Iowa State University)
• Inventors: Skaar, Steven B., Brockman, William H., Yalda-Mooshabod, Issac, Miller, Richard K.
• Primary Examiner(s): Ip, Paul
• Application Number:
  US08/086,749
• Time in Patent Office: 278 Days
• Field of Search: 318/139, 318/16, 318/560-646, 901/1, 901/3, 901/5, 901/7, 901/9, 901/12, 901/13, 901/15-23, 901/45-48, 395/80-99, 180/167-169, 414/730, 414/731, 414/732, 414/735, 250/203 R, 364/478, 364/424.02
• US Class Current: 318/587
• CPC Class Codes: B25J 13/089 Determining the position of the robot with reference to its environment
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B62D 65/00 Designing, manufacturing, e.g. assembling, facilitating disassembly, or structurally modifying motor vehicles or trailers, not otherwise provided for
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means

Abstract

A nonholonomic camera space manipulation system which allows a mobile manipulator which can have both holonomic and nonholonomic movement to autonomously use computer vision to move with respect to a goal position without any prior knowledge or calibration between the two or more video cameras used with the computer vision and the manipulator base or arm, the cameras and the goal, or the base and arm and the goal position. Cues are associated with the manipulator arm and the goal position(s) or target bodies; the cues being distinguishable in the two dimensional focal plane camera spaces of the cameras from the surrounding environment. A processing unit, identifying the position of the cues in each camera space, compares those positions and instructs movements of the manipulator base and/or manipulator arm to achieve movement of the visual cue on the arm with respect to the goal position or target body visual cue in camera space. These instructions are based upon estimations of what movement is required to bring two visual cues in each camera space together or into the required camera-space configuration. Once movement of the manipulator base and arm has begun, the system tracks this movement and adjusts its estimations and instructions according to monitoring of the visual cues in the camera spaces. By utilizing the different angles of the two or more camera means, the continuous monitoring of the visual cue positions in each camera space leads

GOVERNMENT RIGHTS

This invention was made with government support under government contract No. N00014-89-J-1533 award by the U.S. Office of Naval Research. The government has certain rights in the invention.

Claims

1. A camera space manipulation means comprising:

a multiple degree of freedom of movement manipulator means including;

a base means,

an end effector means,

connection means between the base means and the end effector means allowing multiple degrees of freedom of movement of the end effector means with respect to the base means based on known nominal kinematics, and including first sensing means for producing signals correlated to the orientation of the connection means relative to the base means,

motive means associated with the base means for allowing locomotion of the base means in a plurality of degrees of freedom of movement and including second sensing means for producing signals correlated to the distance and direction of movement of the base means along a surface;

first cue means associated with the end effector means;

second cue means associated with a work object;

two or more camera means each having a field of view being positionable to capture at least intermittently both the first and second cue means in a field of view, each camera means without a pre-known positional relationship to the base means, end effector means, connection means or motive means;

camera space means associated with each camera means to convert the field of view of each camera means into a corresponding two dimensional camera space; and

processing means including;

distinguishing means to distinguish the first and second cue means from generally all other contents of the camera spaces,

tracking means to obtain and store information relating to position and movement of the cue means in the camera spaces, monitoring means to obtain and store information relating to holonomic orientation and position of the connection means relative to the base means from the first sensing means, and nonholonomic history of movement of the base means from the second sensing means;

estimation and planning means to repetitively propose a plan of movement for one or both of the connection means and motive means to bring about a desired positional relationship between the first and second cue means in the camera spaces, the plan being based on information in the tracking means and monitoring means taking into consideration both holonomic and nonholonomic relationships,

control means for instructing movements of one or both of the connection and motive means to follow the plan in physical space.

2. The means of claim 1 wherein the base means comprises means for allowing mobility over a surface.

3. The means of claim 1 wherein the base means comprises means for allowing three-dimensional motion or orientation.

4. The means of claim 2 wherein the means for allowing mobility over a surface comprises wheel means.

5. The means of claim 4 wherein the wheel means comprises a plurality of wheels and one or more wheels can be independently driven and one or more wheels can be independently steered.

6. The means of claim 5 wherein at least one wheel is a steerable wheel and a drive wheel.

7. The means of claim 1 wherein the end effector means comprises a grasping means.

8. The means of claim 1 wherein the connection means includes joints and arm segments between the base means and the end effector means.

9. The means of claim 1 wherein the motive means includes a drive motor and a steering motor.

10. The means of claim 1 wherein the motive means includes one or more drive motors and one or more steering motors.

11. The means of claim 1 wherein the visual cue means include a property allowing the visual cue means to be visually distinguishable in camera space.

12. The means of claim 1 wherein the camera means comprises a video camera with image-digitization capability.

13. The means of claim 1 wherein the distinguishing means comprises a contrast detection means.

14. The means of claim 1 wherein the distinguishing means comprises a color differentiation means.

15. The means of claim 1 wherein the distinguishing means comprises a shape differentiation means.

16. The means of claim 1 wherein the distinguishing means comprises a feature differentiation means.

17. The means of claim 1 wherein the tracking means includes a memory means to store a record of position of the first and second cue means.

18. The means of claim 1 wherein the estimation and planning means includes a computer means.

19. The means of claim 1 wherein the control means includes a computer means.

20. The method of nonholonomic camera space manipulation for autonomous manipulation of a multiple-degree-of-freedom manipulator including a base which is mobile across a surface or in three dimensions, a manipulator arm which is manipulable in a plurality of degrees of freedom of movement from the base, and an end effector means at the outer free end of the manipulator arm, comprising:

positioning a visual cue means on at least the end effector means and on a work object means;

positioning two or more camera means so that each field of view of the camera means captures, at least intermittently, the first and second cue means;

communicating the camera means to a control means where the fields of view are digitized and represent two dimensional camera space image planes of the fields of view;

identifying the visual cues in the camera spaces;

estimating movement of the base, manipulator arm, and the end effector means to effect bringing the visual cue means into a desired orientation in each camera space where the estimated movement is based on camera space relationships and on holonomic and nonholonomic considerations related to movement of the manipulator arm, end effector means, and base;

instructing movement of the manipulator arm and base to follow the estimated movement;

monitoring movement of the cues in the camera spaces, holonomic movement of the manipulator arm and end effector, and nonholonomic movement history of the base; and

sequentially updating the estimated movement and instructions based on changes in position in the camera spaces between the visual cues and on the monitoring of movement of cues, manipulator arm, and base to effectuate a desired orientation of cues in the camera spaces and a desired orientation of end effector, manipulator arm, and base in physical space.

21. The method of claim 20 further comprising some holonomic movement of the manipulator arm.

22. The method of claim 20 wherein one visual cue means is placed on the end effector means.

23. The method of claim 20 wherein a visual cue means is placed on an object held by the end effector means.

24. The method of claim 20 wherein the visual cue means comprise one or more visual cue elements.

25. The method of claim 20 wherein the visual cue means comprise a combination of visual cue elements and natural features of the manipulator and work objects.

26. The method of claim 20 wherein the camera means includes means to adjust the camera means.

27. The method of claim 26 wherein the means to adjust the camera means include one or more of servoable zoom means for adjusting the field of view and closeness of view of the camera means, and servoable repositioning means for moving the position and orientation of the camera means.

28. The method of claim 20 wherein the visual cue means are identified from one another in each field of view.

29. The method of claim 20 further comprising computing parameters for movement to arrive at estimates of how the manipulator means associated with one visual cue means should be moved with respect to the other visual cue means on the work object means.

30. The method of claim 20 comprising the further step of defining admissible configurations in camera space for the two visual cue means.

31. The method of claim 20 further comprising defining termination coordinates in camera space for the visual cue means.

32. The method of claim 20 further comprising computing uncertainty of view variables.

33. The method of claim 32 wherein computing uncertainty of view variables involves one or more of the set consisting of algebraic and differential kinematics.

34. The method of claim 20 further comprising creating a mathematical model of movement of the manipulator means.

35. The method of claim 20 further comprising defining termination coordinates for the position of visual cue means with respect to one another.

36. The method of claim 20 further comprising defining orientation of the manipulator means with respect to the work object at a termination position.

37. The method of claim 20 further comprising sequentially updating the estimations in time.

38. The method of claim 20 comprising both holonomic and nonholonomic manipulation.

39. A camera space manipulation control means, utilizing two or more camera means for engaging an end effector means with a work object where there is not required any known prior three-dimensional physical space relationship between the end effector means, the work object, and the camera means, or between physical space and the two dimensional images at image planes, denoted as camera spaces, of the camera means comprising:

an articulateable manipulator means of known nominal kinematics in physical space extending from a mobile base to an outward end for movement of the outward end in a predefined physical work space in the physical space relative to the mobile base, which has a nonholonomic kinematic relationship between wheel rotation and base-position response, the manipulator means including a motor means for articulating the manipulator means in said physical space, the mobile base having a motor means and a steering means to locomote the mobile base in any direction over a surface or in three dimensions, and means for producing a signal identifying an approximate position and orientation of the manipulator means with respect only to the base, wherein the kinematic description of the manipulator means with base being known and the kinematic description of the mobile base being known only relative to prior movement;

each camera means being positionable in physical space without any previously know relation and correlation to the manipulator means except that each camera means must be oriented towards at least the end effector means and the work object for providing camera vision at least intermittently of the end effector means and the work object in camera space;

first visual cue means associated with the end effector means;

second visual cue means associated with the work object, the first and second visual cue means comprising means which are distinct and identifiable in said camera spaces from the remainder of the camera space manipulation control means in any surrounding environment, the first and second visual cue means providing descriptions of three dimensional physical space maneuver objectives as admissible configurations of visual cue means in the two dimensional camera spaces of the camera means; and

a control means operatively connected to the manipulator means and the camera means, the control means including computing means for receiving the signal from the manipulator means and identifying the approximate position and orientation of the manipulator means with respect to the base means through the use of previously known kinematics, and signal processing means which identifies and tracks the visual cue means in the camera spaces to convert such into two dimensional camera space cue position signals, the manipulator approximate position and orientation signal and the camera space cue position signals being used in the control means to estimate the relationship between the position and orientation of the manipulator means and the location in each camera space of the visual cue means placed on the manipulator means, and using the current estimations of these relationships selecting required movement and orientation of the manipulator means which will bring about admissible configurations of the visual cue means in each camera space to insure successful engagement of the object in physical space, and to control orientation of the manipulator means in physical space according to selective movement and orientation commands resulting from the estimated relationship.

40. A method of camera space manipulation utilizing at least two camera means for engaging an articulatable manipulator means with an object where there is not required any known prior three dimensional physical space relationship between the manipulator means and the object, the camera means, and between physical space in a two dimensional image at the focal plane of the camera means, denoted as camera space, comprising the steps:

orienting each camera means to view the manipulator means which has an arm extending from a base to an outward end which is moveable in physical work space with known nominal kinematics relative to the base;

the manipulator means including a motor means which articulates the manipulator means in said physical work space, and means for producing a signal identifying the approximate position and orientation of the manipulator means with respect only to the base in said physical work space;

the base having motor and steering means for moving the base in any direction along a surface and including means for producing a signal identifying the approximate position and orientation of the base with respect only to its prior positions, each camera means being positioned and oriented in physical space without any previously known relation in correlation to the manipulator means except that each camera means must provide, at least intermittently, camera vision of at least the outward end of the manipulator means in at least part of the physical work space to view at least the outer end of the manipulator means and the work object in camera space;

placing a first visual cue means in association with an outward end of the arm;

placing a second visual cue means in association with the object to be engaged by the manipulator means, the first and second visual cue means comprising means which are distinct and identifiable in said camera space from the remainder of the system and any surrounding environment, the first and second visual cue means providing descriptions of three dimensional physical space maneuver objectives in terms of admissible configurations of the visual cue means in the two dimensional camera space of each camera;

receiving signals from the manipulator means and base means and identifying the approximate position and orientation of the manipulator means and base means with respect to the base and surface respectively through the use of known nominal kinematics;

identifying and tracking the visual cue means in the two dimensional camera space of each camera means and repetitively estimating the relationship between the position and orientation of the manipulator means and the location in each camera space of the visual cue means placed on the manipulator means, and using the current estimation of these relationships to select the movement and to command the orientation of the manipulator means which will bring about the admissible configurations of the visual cue means in each camera space which insures successful engagement of the object; and

continuously controlling movement and orientation of the manipulator means according to such autonomously selected movement and orientation commands to achieve engagement of the manipulator means with the work object in said physical work space.

41. A method of nonholonomic camera space manipulation comprising:

initially positioning a multiple-degree-of-freedom-of-movement manipulator means in a three dimensional physical work space;

the manipulator means including a motive means for moving the base means in any direction and orientation in at least the work space;

the manipulator means also including at least one end effector means which is moveable with respect to the base means by a manipulator arm having known kinematics;

initially positioning two or more video camera means so that each field of view of each camera means at least intermittently captures the end effector of the manipulator means in the physical work space;

placing at least one visual cue means which is distinguishable in the field of view in each camera means from all else in the field of view on the end effector means and on a work object that at least temporarily enters the work space;

transforming each field of view of the camera means into a two dimensional camera space having a coordinate system so that any visual cue means captured in the camera space is detected and can be identified according to a coordinate position in the camera space;

monitoring the work space with the camera means until at least temporarily the visual cue means of the end effector means and the visual cue means of the work object are concurrently in one field of view;

comparing the camera space locations of the visual cues;

estimating required movement of the base and manipulator arm according to known kinematics, to move the visual cue in camera space to a predetermined position relative the visual cue of the work object and camera space;

estimating the trajectory of the first and second visual cues and camera space; and

adjusting movement of the base and manipulator arm according to a current estimate of trajectory of the first and second view visual cues in camera space, known arm relative to the base, and history of movement of the base until the desire configuration is achieved.

42. The means of claim 41 wherein the estimation of movement is computed using algebraic and differential calculations.