Situational awareness for teleoperation of a remote vehicle

• US 8,989,876 B2
• Filed: 05/09/2014
• Issued: 03/24/2015
• Est. Priority Date: 02/17/2010
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Lenser, Scott Raymond, Moseley, Mark Bourne
• Primary Examiner(s): Jen, Ian
• Application Number:
  US14/274,227
• Publication Number:
  US 20140247261A1
• Time in Patent Office: 319 Days
• Field of Search: 700/1, 700/2, 700/245, 700/248, 700/250, 700/252
• US Class Current: 700/2
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

A method for improving situational awareness for teleoperation of a remote vehicle by creating a 3D map display of an area around the remote vehicle comprises: receiving an original image from a stereo vision camera and utilizing the original image to perform visual odometry to determine the x, y, z, roll, pitch, and yaw for the original image; applying a fill-in algorithm to the original image to fill in an estimated depth for areas of the original image for which no depth data is available, which creates an enhanced depth image; combining the enhanced depth image with the x, y, z, roll, pitch, and yaw for the original image to create the 3D map display of the area around the remote vehicle; and displaying the 3D map display on an operator control unit used to control the remote vehicle.

Claims

1. A system for providing improved situational awareness for teleoperation of a remote vehicle having a chassis, the system comprising:

a remote vehicle having a range data payload including an imaging device mounted to obtain an unobstructed view of the remote vehicle'"'"'s local environment, the imaging device providing data for construction of a 3D image; and

an operator control unit including a display screen displaying the 3D image of an area around the remote vehicle, the 3D image being constructed from a raw range data produced by the imaging device.

2. The system of claim 1, wherein the operator control unit provides an operator with a virtualized 3D view of the remote vehicle and the pose of the remote vehicle within the local environment surrounding the remote vehicle, as well as spatial representations of obstacles around the remote vehicle.

3. The system of claim 1, wherein the operator control unit further comprises a point-and-click interface on the 3D display of the remote vehicle in the local environment, the point-and-click interface facilitating command and control of the remote vehicle.

4. The system of claim 3, wherein the point-and click includes click and drag operations wherein the display includes a user interface allowing an operator to drive the remote vehicle by clicking on an arrow to cause the remote vehicle to move in a direction indicated by the arrow.

5. The system of claim 4, wherein the user interface includes selectable curved arrows enabling the operator to pivot and pan the remote vehicle and selectable arrows that enable the operator to drive the remote vehicle forward and backward.

10. The system of claim 1, further comprising a computational payload, the computational payload converting the raw data to a 3D map display on the operator control unit by constructing a virtual view of the 3D data directly from the imaging device to create an interactive display.

11. The system of claim 1, wherein the interactive 3D map display includes a local obstacle map constructed using range data provided by the imaging device.

6. The system of clam 1, wherein the remote vehicle further comprises a manipulator arm and the operator control unit provides an operator with a virtualized 3D view of the pose of the manipulator arm.

7. The system of claim 6, wherein instructions for moving the manipulator arm are sent from the operator control unit.

8. The system of claim 6, wherein the manipulator arm further comprises a gripper.

9. The system of claim 6, wherein the operator control unit further comprises a point-and-click interface on the 3D display of the manipulator arm such that selecting the manipulator arm on the 3D map display and dragging it to a desired pose or position causes the remote vehicle to execute the motions required to achieve the desired pose or position.