Method and system for providing autonomous control of a platform

• US 8,229,595 B2
• Filed: 06/19/2009
• Issued: 07/24/2012
• Est. Priority Date: 06/19/2008
• Status: Expired due to Fees

• Current Assignee: John-David Yoder, Michael J. Seelinger
• Original Assignee: John-David Yoder, Michael J. Seelinger
• Inventors: Seelinger, Michael J., Yoder, John-David
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): Amin, Bhavesh V
• Application Number:
  US12/487,757
• Publication Number:
  US 20100234993A1
• Time in Patent Office: 1,131 Days
• Field of Search: None
• US Class Current: 700/254
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/37567 3-D vision, stereo vision, with two cameras
• CPC Class Codes: G05B 2219/39397 Map image error directly to robot movement, position with relation to world, base not needed, image based visual servoing
• CPC Class Codes: G05B 2219/40003 Move end effector so that image center is shifted to desired position
• CPC Class Codes: G05B 2219/40298 Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593

Abstract

The present application provides a system for enabling instrument placement from distances on the order of five meters, for example, and increases accuracy of the instrument placement relative to visually-specified targets. The system provides precision control of a mobile base of a rover and onboard manipulators (e.g., robotic arms) relative to a visually-specified target using one or more sets of cameras. The system automatically compensates for wheel slippage and kinematic inaccuracy ensuring accurate placement (on the order of 2 mm, for example) of the instrument relative to the target. The system provides the ability for autonomous instrument placement by controlling both the base of the rover and the onboard manipulator using a single set of cameras. To extend the distance from which the placement can be completed to nearly five meters, target information may be transferred from navigation cameras (used for long-range) to front hazard cameras (used for positioning the manipulator).

Claims

1. A method for operating an autonomous vehicle that includes a manipulator and one or more sets of cameras on the autonomous vehicle, the method comprising:

calibrating the manipulator with the one or more sets of cameras to establish calibration parameters describing a relationship between a location of features of the manipulator in a two-dimensional image acquired by the one or more sets of cameras and a three-dimensional position of the features of the manipulator, wherein the one or more sets of cameras comprises a first set of cameras and a second set of cameras;

determining a first camera-space target projection based on a relationship between a three-dimensional location of a target and a location of the target in a given two-dimensional image acquired by the first set of cameras;

using the calibration parameters and the first camera-space target projection to estimate a location of the target relative to the manipulator;

creating a trajectory for the autonomous vehicle and the manipulator to follow to position the autonomous vehicle and the manipulator such that the manipulator can engage the target based on the three-dimensional location of the target due to the first camera-space target projection in the relationship between the three-dimensional location of the target and the location of the target in the given two-dimensional image acquired by the first set of cameras;

updating the first camera-space target projection based on subsequent two-dimensional images acquired by the first set of cameras as the autonomous vehicle traverses the trajectory;

based on a distance of the autonomous vehicle to the target, transitioning the target from the first set of cameras to the second set of cameras;

determining a second camera-space target projection based on a relationship between a three-dimensional location of the target and a location of the target in a given two-dimensional image acquired by the second set of cameras; and

updating the trajectory for the autonomous vehicle and the manipulator to follow based on the second camera-space target projection,

wherein transitioning the target from the first set of cameras to the second set of cameras comprises;

using the given two-dimensional image acquired by the first set of cameras, providing a laser onto the target;

receiving the given two-dimensional image of the target acquired by at least one of the second set of cameras; and

determining a camera-space location of the laser in the given two-dimensional image acquired by the second set of cameras based on a location of the laser in the image.

2. The method of claim 1, wherein the first set of cameras on the autonomous vehicle are long-range viewing cameras and the second set of cameras on the autonomous vehicle are short-range viewing cameras.

3. The method of claim 1, wherein transitioning the target from the first set of cameras to the second set of cameras comprises:

positioning a light source to emit a light substantially near the location of the target;

at least one of the second set of cameras acquiring an image including the light;

updating the three-dimensional location of the target based on a location of the light in the image.

4. The method of claim 1, further comprising determining a configuration of the system to position the manipulator approximately at the three-dimensional location of the target.

5. The method of claim 1, further comprising:

positioning a light source to emit a light substantially near the location of the target;

identifying the light within images produced by one of the first set of cameras and the second set of cameras; and

defining the location of the target in the images produced by one of the first set of cameras and the second set of cameras.

6. The method of claim 1, wherein calibrating comprises:

moving the manipulator through a series of positions;

at each positions, acquiring images from the cameras and recording corresponding angles and positions of the manipulator;

in each image, identifying a camera-space location of a feature of the manipulator;

establishing parameters describing a relationship between the camera-space location of the feature of the manipulator and a three-dimensional position of the feature of the manipulator.

7. The method of claim 1, wherein determining a relationship between a three-dimensional location of a target and a location of the target in a given two-dimensional image acquired by the first set of cameras comprises using the CAHVOR camera model.

8. The method of claim 7, wherein the CAHVOR camera model includes six vectors of three parameters each for a total of eighteen camera model parameters comprising c={c0,c1,c2}, a={a0,a1,a2}, h={h0,h1,h2}, v={v0,v1,v2}, o={o0,o1,o2}, r={r0,r1,r2}, where the location of the target in the two-dimensional image is described with coordinates (x, y) and the three-dimensional location of the target is described by vector p, and where

9. The method of claim 1, wherein creating the trajectory for the autonomous vehicle and the manipulator to follow comprises:

determining a position relative to the autonomous vehicle at which to move the manipulator;

determining coordinates of the location of the target; and

determining a given trajectory for the autonomous vehicle and instructions for following the trajectory.

10. The method of claim 1, further comprising:

receiving a second two-dimensional image acquired by the one or more sets of cameras; and

updating the calibration parameters to describe a relationship between a location of features of the manipulator in the second two-dimensional image and a current three-dimensional position of the features of the manipulator.

11. A method for operating an autonomous vehicle that includes a manipulator and one or more sets of cameras on the autonomous vehicle, the method comprising:

calibrating the manipulator with the one or more sets of cameras to establish calibration parameters describing a relationship between a location of features of the manipulator in a two-dimensional image acquired by the one or more sets of cameras and a three-dimensional position of the features of the manipulator, wherein the one or more sets of cameras comprises a first set of cameras and a second set of cameras;

determining a first camera-space target projection based on a relationship between a three-dimensional location of a target and a location of the target in a given two-dimensional image acquired by the first set of cameras;

using the calibration parameters and the first camera-space target projection to estimate a location of the target relative to the manipulator;

creating a trajectory for the autonomous vehicle and the manipulator to follow to position the autonomous vehicle and the manipulator such that the manipulator can engage the target based on the three-dimensional location of the target due to the first camera-space target projection in the relationship between the three-dimensional location of the target and the location of the target in the given two-dimensional image acquired by the first set of cameras;

updating the first camera-space target projection based on subsequent two-dimensional images acquired by the first set of cameras as the autonomous vehicle traverses the trajectory;

based on a distance of the autonomous vehicle to the target, transitioning the target from the first set of cameras to the second set of cameras;

determining a second camera-space target projection based on a relationship between a three-dimensional location of the target and a location of the target in a given two-dimensional image acquired by the second set of cameras; and

updating the trajectory for the autonomous vehicle and the manipulator to follow based on the second camera-space target projection,

wherein creating the trajectory for the autonomous vehicle and the manipulator to follow comprises;

using the first camera-space target projection when the distance between the autonomous vehicle and the target is above a threshold distance; and

using the second camera-space target projection when the distance between the autonomous vehicle and the target is below a threshold distance.