Situational Awareness for Teleoperation of a Remote Vehicle

• US 20110264303A1
• Filed: 02/17/2011
• Published: 10/27/2011
• Est. Priority Date: 02/17/2010
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: LENSER, Scott Raymond, Moseley, Mark Bourne
• Granted Patent: US 8,725,273 B2
• Time in Patent Office: Days
• US Class Current: 701/2
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

A method for improving situational awareness for teleoperation of a remote vehicle by creating a 3D map display of an area around the remote vehicle comprises: receiving an original image from a stereo vision camera and utilizing the original image to perform visual odometry to determine the x, y, z, roll, pitch, and yaw for the original image; applying a fill-in algorithm to the original image to fill in an estimated depth for areas of the original image for which no depth data is available, which creates an enhanced depth image; combining the enhanced depth image with the x, y, z, roll, pitch, and yaw for the original image to create the 3D map display of the area around the remote vehicle; and displaying the 3D map display on an operator control unit used to control the remote vehicle.

Claims

1. A method for improving situational awareness for teleoperation of a remote vehicle by creating a 3D map display of an area around the remote vehicle, the method comprising:

receiving an original image from a stereo vision camera and utilizing the original image to perform visual odometry to determine the x, y, z, roll, pitch, and yaw for the original image;

applying a fill-in algorithm to the original image to fill in an estimated depth for areas of the original image for which no depth data is available, which creates an enhanced depth image;

combining the enhanced depth image with the x, y, z, roll, pitch, and yaw for the original image to create the 3D map display of the area around the remote vehicle; and

displaying the 3D map display on an operator control unit used to control the remote vehicle.

2. The method of claim 1, further comprising displaying the 3D map and a primary video feed adjacent each other.

3. The method of claim 2, further comprising allowing the operator to turn the 3D map display on and off.

4. The method of claim 3, further comprising displaying the 3D map display as a supplemental operator display choice rather than replacing an existing operator display choice.

5. A system for providing improved situational awareness for teleoperation of a remote vehicle having a chassis, the system comprising:

a remote vehicle having a range data payload and a computational payload, the computation payload providing a visual odometry map construction for a 3D image; and

an operator control unit including a display screen displaying the 3D image of an area around the remote vehicle.

6. The system of claim 5, wherein the 3D image is constructed from raw stereo vision data.

7. The system of claim 5, wherein the 3D image is constructed from raw stereo vision data, visual odometry, fill-in algorithms, and plane fitting.

8. The system of claim 5, wherein the range data payload comprises a SEER payload.

9. The system of claim 8, wherein the computation payload comprises a Navigator payload.

10. The system of claim 9, further comprising a manipulator arm payload.

11. The system of claim 10, wherein the remote vehicle chassis communicates with the Navigator payload via Ethernet and provides power to the Navigator payload,

wherein the remote vehicle chassis also communicates with the SEER payload via Ethernet and provides power to the SEER payload, and

wherein the manipulator arm payload plugs into the SEER payload and the remote vehicle chassis provides power to the manipulator arm payload via the SEER payload.

12. The system of claim 11, wherein the manipulator arm comprises a camera, and analog video from the camera is passed to the chassis via the SEER payload.

13. The system of claim 12, wherein operator instructions for the manipulator arm payload are passed to the manipulator arm payload via instructions sent from the operator control unit to the remote vehicle chassis and from the remote vehicle chassis to the manipulator arm payload via the SEER payload.

14. The system of claim 10, wherein the Navigator payload attaches to a payload bay on the remote vehicle in a plug-and-play fashion, the SEER payload attaches to a payload bay on the remote vehicle in a plug-and-play fashion, and the manipulator arm payload plugs into the SEER payload in a plug-and-play fashion.

15. A system for providing improved situational awareness for teleoperation of a remote vehicle having a chassis, the system comprising:

a remote vehicle having a range data payload and a computational payload, and a manipulator arm payload, the computation payload providing a visual odometry map construction for a 3D image; and

an operator control unit including a display screen displaying the 3D image of an area around the remote vehicle, the 3D image being constructed from raw stereo vision data, visual odometry, fill-in algorithms, and plane fitting.

16. The system of claim 15, wherein, in the plane fitting algorithm:

an image from the stereo vision camera is segmented and a plane is fit to the image segment;

a depth image is used as input to the plane fitting step;

the segmented image includes the image and a list of the segment that each pixel belongs to, including depth;

the algorithm fills in image depth from the plane if the plane fitting step is successful, stores the depth image, and determines whether all of the image segments have been processed the next segment is plane fitted and processed as described above if the plane fitting step is not successful.

17. The system of claim 16 wherein in the plane-fitting algorithm:

an image segment is dilated;

the dilated image segment and a depth image are used to determine location relative to the stereo vision camera and collect 3D points;

a Random Sample Consensus (RANSAC)-based algorithm is used to fit a first plane to the points;

the fit of the first plane to the points is checked based on the number of points that are sufficiently close the first plane;

the system fails if the first plane fit is not okay;

the algorithm tries to fit a line to the points if the first plane fit is okay;

the first plane is replaced by a constrained plane and plane fit is checked again;

the final plane is the constrained plane and the system is finished plane fitting if a line can be fit to the points of the constrained plane;

the final plane is the first plane and the system is finished plane fitting if a line cannot be fit to the points of the constrained plane.