METHODS, SYSTEMS, AND APPARATUS FOR MULTI-SENSORY STEREO VISION FOR ROBOTICS

• US 20160227193A1
• Filed: 03/14/2014
• Published: 08/04/2016
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Uatc, LLC (Uber Technologies, Inc.)
• Original Assignee: Uber Technologies, Inc.
• Inventors: Osterwood, Christopher Charles, Strother, Daniel Leland, LaRose, David Arthur
• Granted Patent: US 10,412,368 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 7/4972 Alignment of sensor
• CPC Class Codes: G03B 17/02 Bodies
• CPC Class Codes: G03B 17/55 with provision for heating or cooling, e.g. in aircraft
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Embodiments of the present invention include multi-sensory stereo vision sensors suitable for use in robotics, navigation, machine vision, manufacturing, and other applications. In some cases, a sensor includes a stereo camera that produces image data for use in generating disparity maps to determine the positions of objects in the scene and/or the position of the sensor itself. The stereo sensor may include image sensors that are fastened to a unitary frame to prevent undesired drift and a thermal pad that wicks heat away from a processor. The processor may provide an efficient means to directly compute stereo disparity maps onboard the sensor. A sensor can also include a laser rangefinder that provides range data suitable for calibrating the stereo camera and for improving its accuracy in certain environments. In some cases, the laser is coupled to a spindle, which in turn is driven by a geartrain through a slipping clutch.

Claims

1. A system for estimating a position of at least one object in an environment, the system comprising:

a first imager to acquire at least two reference images of the environment from a first perspective;

a second imager to acquire at least two comparative images of the environment from a second perspective; and

a processor, operably coupled to the first imager and the second imager, to;

(i) synthesize a first disparity map of the environment from a first reference image in the at least two reference images and a first comparative image in the at least two comparative images;

(ii) synthesize a second disparity map of the environment from a second reference image in the at least two reference images and a second comparative image in the at least two comparative images;

(iii) determine a first position estimate of the position of the at least one object based at least in part on the first disparity map; and

(iv) determine a second position estimate of the position of the at least one object based at least in part on the second disparity map.

2. The system of claim 1, wherein the processor is further configured to estimate a difference between the first position estimate and the second position estimate.

3. The system of claim 2, wherein the processor is further configured to correct for the difference between the first position estimate and the second position estimate.

4. The system of claim 3, further comprising:

a laser to illuminate the at least one object in the environment with laser light; and

a sensor to detect laser light scattered and/or reflected by the at least one object and to provide a range signal representative of a distance to the object based on the detected laser light,

wherein the processor is further configured to determine the difference between the first position estimate and the second position estimate by comparing the range signal to at least one of the first position estimate and the second position estimate.

5. The system of claim 1, wherein the at least one object comprises the system.

6. The system of claim 1, wherein the at least one object comprises an object other than the system.

7. A method of calibrating a multi-sensory stereo vision system comprising a stereo imager and a rangefinder, the method comprising:

(A) determining, from a disparity map provided by the stereo imager, a first three-dimensional position estimate of at least one object appearing in the disparity map;

(B) determining, from a measurement by the rangefinder, a second three-dimensional position estimate of the at least one object;

(C) determining a difference between the first three-dimensional position estimate determined in (A) and the second three-dimensional position estimate determined in (B); and

(D) calibrating the multi-sensory stereo vision system based on the difference determined in (C).

8. The method of claim 7, wherein (A) further comprises:

(A1) acquiring a first two-dimensional image of a scene viewed from a first perspective by the stereo imager;

(A2) acquiring a second two-dimensional image of the scene viewed from a second perspective by the stereo imager; and

(A3) synthesizing the disparity map from the first two-dimensional image and the second two-dimensional image.

9. The method of claim 7, wherein the rangefinder comprises a laser rangefinder and wherein (B) further comprises:

(B1) illuminating the at least one object with laser light;

(B2) detecting laser light scattered and/or reflected by the at least one object; and

(B3) determining the second three-dimensional position based at least in part on the laser light detected in (B2).

10. The method of claim 7, wherein (C) further comprises:

(C1) projecting coordinates of the second three-dimensional position estimate onto a volume defined by the disparity map; and

(C2) determining a distance between the coordinates of the second three-dimensional position estimate and coordinates of the first three-dimensional position estimate.

11. A method of calibrating a multi-sensory stereo vision system comprising a stereo imager and a rangefinder, the method comprising:

(A) acquiring a first two-dimensional image of a scene viewed from a first perspective by the stereo imager;

(B) acquiring a second two-dimensional image of the scene viewed from a second perspective by the stereo imager;

(C) synthesizing the disparity map from the first two-dimensional image acquired in (A) and the second two-dimensional image acquired in (B);

(D) determining, from a disparity map synthesized in (C), a first three-dimensional position estimate of at least one object appearing in the disparity map;

(E) illuminating the at least one object with laser light;

(F) detecting laser light scattered and/or reflected by the at least one object illuminated in (E);

(G) determining a second three-dimensional position estimate of the at least one object based at least in part on the laser light detected in (F);

(H) projecting the second three-dimensional position estimate determined in (G) onto a volume defined by the disparity map determined in (C);

(I) determining a difference between the second three-dimensional position estimate projected in (H) and the first three-dimensional position estimate;

(J) calibrating the multi-sensory stereo vision system based on the difference determined in (I).

12. A system for imaging at least one feature in an environment, the system comprising:

a first imager to acquire at least two reference images of the environment from a first perspective;

a second imager to acquire at least two comparative images of the environment from a second perspective; and

a processor, operably coupled to the first imager and the second imager, to;

(i) synthesize a first disparity map of the environment from a first reference image in the at least two reference images and a first comparative image in the at least two comparative images;

(ii) synthesize a second disparity map of the environment from a second reference image in the at least two reference images and a second comparative image in the at least two comparative images;

(iii) identify the at least one feature in the first disparity map;

(iv) identify the at least one feature in the second disparity map; and

(v) estimate a difference in appearance of the at least one feature between the first disparity map and the second disparity map.

13. A system for detecting misalignment of a scanning beam of laser light, the system comprising:

a laser to provide a laser beam;

a scanner, in optical communication and/or mechanical association with the laser, to scan the laser beam so as to provide the scanning beam of laser light;

a sensor to detect laser light scattered and/or reflected by at least one object in the environment and to provide a first signal representative of a first measurement of the detected laser light and a second signal representative of a second measurement of the detected laser light; and

a processor, operably coupled to the sensor, to;

(i) generate a first representation of the environment based on the first signal;

(ii) generate a second representation of the environment based on the second signal;

(iii) estimate at least one difference between the first representation and the second representation; and

(iv) determine a misalignment of at least one of the laser, the scanner, and the sensor based on the at least one difference between the first representation and the second representation.

14. The system of claim 13, wherein the processor is further configured to account for movement of at least one of the laser, the scanner, and the sensor between the first measurement and the second measurement in determining the misalignment of the at least one of the laser, the actuator, and the sensor.

15. The system of claim 13, wherein the sensor comprises a stereo image sensor and the processor is configured to calibrate the stereo sensor based on the first representation and/or the second representation.

16. The system of claim 15, wherein the processor is configured to estimate a first three-dimensional position of at least one of the laser, the scanner, and the stereo image sensor based at least in part on the first representation.

17. The system of claim 16, wherein the processor is further configured to determine a second three-dimensional position estimate of the at least one of the laser, the scanner, and the stereo image sensor and to estimate the at least one difference based on the first three-dimensional position estimate and the second three-dimensional position estimate.

18. A processing unit for producing a disparity map of an environment based on image data, the processing unit comprising:

at least one interface to receive image data from at least two imagers;

at least one rectification processing block, operably coupled to the at least one interface, to rectify the image data; and

at least one stereo processing block, operably coupled to the at least one rectification processing block, to produce the disparity map based on the image data.

19. The processing unit of claim 18, wherein the at least one interface comprises:

a first interface to transform a first serial representation of a first portion of the image data into a first parallel representation of the first portion of the image data; and

a second interface to transform a second serial representation of a second portion of the image data into a second parallel representation of the second portion of the image data.

20. The processing unit of claim 18, wherein the rectification processing block and/or the stereo processing block are implemented in a field-programmable gate array.

21. The processing unit of claim 18, wherein the at least one stereo processing block is configured to produce the disparity map according to at least one of a semi-global block matching (SGBM) algorithm, a semi-global matching (SGM) algorithm, and a stereo block matching algorithm.

22. The processing unit of claim 18, wherein the rectification block comprises at least one memory-to-memory block.

23. The processing unit of claim 22, wherein the at least one memory-to-memory block comprises:

a frontend to retrieve the at least a portion of the image data from a memory;

a delay line, operably coupled to the frontend, to store at least one image processing command for a period equal to or greater than a latency of the memory; and

a backend, operably coupled to the delay line, to produce a rectified output based at least in part on the image data retrieved from the memory.

24. The processing unit of claim 18, further comprising:

at least one pre-processing block, operably coupled to the at least one interface, to pre-process a parallel representation of the image data.

25. The processing unit of claim 24, wherein the at least one pre-processing block is configured to perform at least one of filtering, histogram generation, linearization, vignette correction, demosaicing, white balancing, and colorspace conversion of the parallel representation of the data.

26. A laser scanning system comprising:

a laser to generate a laser beam;

a spindle, mechanically coupled to the laser, to rotate the laser about a first axis so as to scan the laser beam;

a gear to apply torque to the spindle; and

a slipping clutch friction pad, compressed between the gear and the spindle, to convey at least a portion of the torque applied by the gear to the spindle up to a threshold torque and to allow the gear to slip with respect to the spindle at torques above the threshold torque.

27. The laser scanning system of claim 26, wherein the threshold torque remains constant as a function of temperature and force applied to the spindle.

28. The laser scanning system of claim 26, further comprising:

a housing containing at least a portion of the spindle, the gear, and the slipping clutch friction pad; and

an encoder, mechanically coupled to the spindle, to measure a position of the spindle with respect to the housing.

29. The laser scanning system of claim 26, wherein the gear is a worm gear and further comprising:

a worm, meshed with the gear, to drive the worm gear; and

a motor, mechanically coupled to the worm, to spin the worm so as to drive the worm gear.

30. The laser scanning system of claim 29, further comprising:

a compliant motor mount, mechanically coupled to the motor, to permit axial and radial movement of the motor.

31. The laser scanning system of claim 26, further comprising:

a slip ring mechanically coupled to the spindle and in electrical communication with the laser; and

a power source, in electrical communication with the laser via the slip ring, to power the laser.

32. A stereo vision system comprising:

a unitary frame defining at least one mounting surface;

a first imager, mounted to the at least one mounting surface, to acquire at least one first image of an environment from a first perspective; and

a second imager, mounted to the at least one mounting surface, to acquire at least one second image of the environment from a second perspective different than the first perspective.

33. The stereo vision system of claim 32, wherein the unitary frame consists of a single piece of material.

34. The stereo vision system of claim 32, wherein the first imager and the second imager are disposed in the same plane.

35. The stereo vision system of claim 32, wherein the first imager is disposed in a first plane and the second imager is disposed in a second plane parallel to the first plane.

36. The stereo vision system of claim 32, wherein the first imager is disposed in a first plane and the second imager is disposed in a second plane that intersects the first plane.

37. The stereo vision system of claim 32, further comprising:

at least one of at least one pin and at least one bolt fastening at least one of the first imager and the second imager to the at least one mounting surface.

38. The stereo vision system of claim 32, further comprising:

a processor, operably coupled to the first imager and the second imager, to produce stereo image data from the at least one first image and the at least one second image; and

a flexible cable coupling the second imager to the processor.

39. A sensor system comprising:

at least one sensor to provide sensor data;

a housing defining a cavity and at least one exterior surface;

an internal wall disposed within the cavity and in thermal communication with the at least one exterior surface;

a processor, disposed within the cavity and operably coupled to the at least one sensor, to process the sensor data;

a thermally conductive material, in thermal communication with the internal wall and the processor, to conduct heat generated by the processor to the at least one exterior surface via the internal wall so as to dissipate the heat generated by the processor.

40. The sensor system of claim 39, wherein the at least one sensor is disposed within the cavity.

41. The sensor system of claim 39, wherein the at least one sensor comprises:

a first imager, mounted to at least one mounting surface, to acquire at least one first image of an environment from a first perspective; and

a second imager, mounted to the at least one mounting surface, to acquire at least one second image of the environment from a second perspective different than the first perspective.

42. The sensor system of claim 41, further comprising at least one pin and/or at least one bolt fastening at least one of the first imager and the second imager to the at least one mounting surface.

43. A sensor system comprising:

at least one sensor to provide sensor data;

a housing defining a cavity and comprising at least one wall defining at least one exterior surface;

a processor, disposed within the cavity abutting the at least one wall and operably coupled to the at least one sensor, to process the sensor data,

wherein the processor is in thermal communication with the exterior surface via the at least one wall so as to dissipate the heat generated by the processor.

44. A stereo vision system comprising:

a housing;

a first sensor array, disposed within the housing, to acquire at least one first image of the environment from a first perspective via a first aperture in the housing; and

a second sensor array, disposed within the housing, to acquire at least one second image of the environment from a second perspective different than the first perspective via a second aperture in the housing; and

a light source, disposed within the housing, to illuminate an environment outside the housing via a third aperture in the housing.

45. The stereo vision system of claim 44, further comprising:

a first lens, disposed in the first aperture and in optical communication with the first sensor array, to image a first portion of the environment onto the first sensor array; and

a second lens, disposed in the second aperture and in optical communication with the second sensor array, to image a second portion of the environment onto the second sensor array.