Rotating LIDAR with co-aligned imager

• US 10,447,973 B2
• Filed: 08/08/2017
• Issued: 10/15/2019
• Est. Priority Date: 08/08/2017
• Status: Active Grant

• Current Assignee: Waymo LLC (Alphabet Inc.)
• Original Assignee: Waymo LLC (Alphabet Inc.)
• Inventors: Droz, Pierre-Yves, Verghese, Simon, Hermalyn, Brendan
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US15/671,845
• Publication Number:
  US 20190052844A1
• Time in Patent Office: 798 Days
• Field of Search: 348148
• CPC Class Codes: B60R 1/00 Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
• CPC Class Codes: B60R 2300/101 using cameras with adjustable capturing direction
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4812 transmitted and received beams following a coaxial path
• CPC Class Codes: G01S 7/4817 relating to scanning
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Example implementations are provided for an arrangement of co-aligned rotating sensors. One example device includes a light detection and ranging (LIDAR) transmitter that emits light pulses toward a scene according to a pointing direction of the device. The device also includes a LIDAR receiver that detects reflections of the emitted light pulses reflecting from the scene. The device also includes an image sensor that captures an image of the scene based on at least external light originating from one or more external light sources. The device also includes a platform that supports the LIDAR transmitter, the LIDAR receiver, and the image sensor in a particular relative arrangement. The device also includes an actuator that rotates the platform about an axis to adjust the pointing direction of the device.

Claims

1. A device comprising:

a light detection and ranging (LIDAR) transmitter that emits light pulses toward a scene according to a pointing direction of the device;

a LIDAR receiver that detects reflections of the emitted light pulses from the scene;

an image sensor that captures an image of the scene according to the pointing direction of the device and based on at least external light originating from one or more external light sources, wherein the image sensor comprises an array of sensing elements, wherein the image sensor generates an image pixel of the captured image based on a sum of at least;

a first image pixel indicated by a first sensing element in the array, and a second image pixel indicated by a second sensing element in the array;

a platform that supports the LIDAR transmitter, the LIDAR receiver, and the image sensor in a particular relative arrangement; and

an actuator that rotates the platform about an axis to adjust the pointing direction of the device, wherein a time delay between capture of the first image pixel and capture of the second image pixel is based on at least a rate of the rotation of the platform about the axis.

2. The device of claim 1, wherein the controller associates data from the LIDAR receiver with data from the image sensor.

3. The device of claim 1, wherein the controller determines a three-dimensional (3D) representation of the scene based on data from the LIDAR receiver and data from the image sensor.

4. The device of claim 1, wherein the controller determines a representation of the scene based on color information indicated by the image sensor and distance information indicated by the LIDAR receiver.

5. The device of claim 1, wherein the controller causes the image sensor to capture a plurality of images while the actuator is rotating the platform.

6. The device of claim 5, wherein the controller determines a given time delay between capture of two consecutive images by the image sensor based on the rate of rotation of the platform about the axis, and wherein the controller causes the image sensor to capture the plurality of images according to the given time delay.

7. The device of claim 5, wherein the controller causes the image sensor to capture a given image of the plurality of images at a given time that is based on an emission time of one or more of the emitted light pulses.

8. The device of claim 1, further comprising:

a lens that receives light propagating toward the device and incident on the lens, wherein the lens transmits and focuses the received light through the lens, wherein the image sensor images the scene through the lens, and wherein the LIDAR receiver receives the reflections of the emitted light pulses through the lens.

9. The device of claim 8, further comprising:

an optical element positioned along a path of the focused light from the lens, wherein the optical element directs a first portion of the focused light toward the LIDAR receiver, and wherein the optical element directs a second portion of the focused light toward the image sensor.

10. The device of claim 8, further comprising:

a light filter positioned along a path of the focused light from the lens, wherein the light filter attenuates light having wavelengths outside a wavelength range of the emitted light pulses, wherein the LIDAR receiver receives the reflections of the emitted light pulses through the light filter, wherein the image sensor receives at least a portion of the focused light that does not propagate through the light filter.

11. A vehicle comprising:

a light detection and ranging (LIDAR) sensor that includes a transmitter and a receiver, wherein the transmitter emits light pulses toward a scene according to a pointing direction of the LIDAR sensor, and wherein the receiver detects reflections of the emitted light pulses propagating from the scene;

a camera that captures an image according to a pointing direction of the camera and based on at least external light originating from one or more external light sources, wherein the camera generates a column of image pixels in the image based on at least a sum of;

a first image pixel column captured using a first column of sensing elements in the camera, and a second image pixel column captured using a second column of sensing elements in the camera;

a platform that supports the LIDAR sensor and the camera in a particular relative arrangement; and

an actuator that rotates the platform about an axis to simultaneously change the pointing direction of the LIDAR sensor and the pointing direction of the camera according to the particular relative arrangement, and wherein a time delay between capture of the first image pixel column and capture of the second image pixel column is based on at least a rate of rotation of the platform about the axis.

12. The vehicle of claim 11, further comprising a controller that causes the camera to capture a plurality of images according to detection times of one or more reflected light pulses detected by the LIDAR sensor.

13. The vehicle of claim 11, wherein the camera comprises one or more columns of sensing elements substantially parallel to the axis of rotation of the platform.

14. The vehicle of claim 13, wherein the one or more columns of sensing elements includes at least the first column of sensing elements and the second column of sensing elements.

15. The vehicle of claim 11, further comprising:

a LIDAR lens that focuses the reflections of the emitted light pulses propagating from the environment toward a receiver of the LIDAR sensor; and

a camera lens that focuses the external light toward an array of sensing elements in the camera.

16. A method comprising:

scanning a scene using a light detection and ranging (LIDAR) sensor, wherein the LIDAR sensor emits light pulses toward the scene and detects reflections of the emitted light pulses from the scene;

generating an image of the scene using an image sensor that detects external light originating from one or more external light sources, wherein generating the image comprises;

obtaining a first image pixel captured using a first sensing element in the image sensor,

obtaining a second image pixel captured, after passage of a time delay from a time when the first image pixel is captured, using a second sensing element in the image sensor, and

determining a particular image pixel of the generated image of the scene based on at least a sum of the first image pixel and the second image pixel; and

rotating a platform that supports the LIDAR sensor and the image sensor in a particular relative arrangement, wherein rotating the platform comprises simultaneously rotating the LIDAR sensor and the image sensor about an axis, and wherein the time delay is based on at least a rate of rotation of the platform about the axis.

17. The method of claim 16, further comprising:

selecting the second sensing element of the image sensor based on at least distance information associated with the first image pixel and indicated by data from the LIDAR sensor.