Methods, systems, and apparatus for multi-sensory stereo vision for robotics

• US 10,412,368 B2
• Filed: 03/14/2014
• Issued: 09/10/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Aurora Operations Incorporated (Aurora Innovation, Inc.)
• Original Assignee: Uber Technologies, Inc.
• Inventors: Osterwood, Christopher Charles, Strother, Daniel Leland, LaRose, David Arthur
• Primary Examiner(s): Chowdhury, Nigar
• Application Number:
  US14/777,427
• Publication Number:
  US 20160227193A1
• Time in Patent Office: 2,006 Days
• Field of Search: 348 47, 348 42, 348 49
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G03B 17/02 Bodies
• CPC Class Codes: G03B 17/55 with provision for heating or cooling, e.g. in aircraft
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Embodiments of the present invention include multi-sensory stereo vision sensors suitable for use in robotics, navigation, machine vision, manufacturing, and other applications. In some cases, a sensor includes a stereo camera that produces image data for use in generating disparity maps to determine the positions of objects in the scene and/or the position of the sensor itself. The stereo sensor may include image sensors that are fastened to a unitary frame to prevent undesired drift and a thermal pad that wicks heat away from a processor. The processor may provide an efficient means to directly compute stereo disparity maps onboard the sensor. A sensor can also include a laser rangefinder that provides range data suitable for calibrating the stereo camera and for improving its accuracy in certain environments. In some cases, the laser is coupled to a spindle, which in turn is driven by a geartrain through a slipping clutch.

Claims

1. A stereo vision system for an autonomous vehicle, the stereo vision system comprising:

a housing defining a cavity and at least one exterior surface;

a unitary frame within the cavity, the unitary frame defining at least one mounting surface;

a first imager, mounted to the at least one mounting surface, to acquire images of an environment from a first perspective;

a second imager, mounted to the at least one mounting surface, to acquire images of the environment from a second perspective different than the first perspective;

a collection of processing units, the collection of processing units including a processing unit disposed in the housing, wherein the collection of processing units is operable to;

obtain a three-dimensional point cloud of the environment from a LIDAR system;

generate a first disparity map based on images acquired by each of the first imager and the second imager;

calibrate each of the first imager and the second imager, using each of the three-dimensional point cloud and the first disparity map, by determining a similarity between data from the three-dimensional point cloud of the environment and the first disparity map;

generate a second disparity map based on images acquired by the calibrated first imager and the calibrated second imager; and

determine information about the environment using the images acquired by the calibrated first imager, the calibrated second imager, the second disparity map and the three-dimensional point cloud.

2. The stereo vision system of claim 1, wherein the unitary frame consists of a single piece of material.

3. The stereo vision system of claim 1, wherein the first imager and the second imager are disposed on a same plane.

4. The stereo vision system of claim 1, wherein the first imager is disposed in a first plane and the second imager is disposed in a second plane parallel to the first plane.

5. The stereo vision system of claim 1, wherein the first imager is disposed in a first plane and the second imager is disposed in a second plane that intersects the first plane.

6. The stereo vision system of claim 1, further comprising:

at least one of at least one pin and at least one bolt fastening at least one of the first imager and the second imager to the at least one mounting surface.

7. The stereo vision system of claim 1, further comprising:

a flexible cable coupling the second imager to the processing unit.

8. The stereo vision system of claim 1, further comprising:

an internal wall disposed within the cavity and in thermal communication with the at least on exterior surface.

9. The stereo vision system of claim 8, further comprising:

a thermally conductive material that is disposed within the cavity and in thermal communication with the internal wall and the processing unit, to conduct heat generated by the processing unit to the at least one exterior surface via the internal wall so as to dissipate the heat generated by the processing unit.

10. The stereo vision system of claim 1, wherein the collection of processing units includes a field programmable gate array.

11. The stereo vision system of claim 1, wherein the collection of processing units is further operable to:

determine a position of the autonomous vehicle.

12. The stereo vision system of claim 1, wherein determining information about the environment includes determining a position of one or more moving objects within the environment.

13. The stereo vision system of claim 1, further comprising:

a rectification processing block to rectify the images from the first imager and the second imager.

14. The stereo vision system of claim 1, wherein the collection of processing units includes logic to separately trigger signals to the first imager and to the second imager so that the signals are received at the first imager and second imager at the same time.

15. The stereo vision system of claim 1, wherein the LIDAR system includes:

a laser mounted on top of the housing to illuminate one or more objects in the environment with laser light; and

a sensor to detect laser light scattered and/or reflected by the one or more objects and to provide a range signal representative of a distance to the one or more objects based on the detected laser light, wherein the three-dimensional point cloud is based on the range signal representative of the distance to the one or more objects.