Methods and Apparatus for Demultiplexing Illumination

• US 20150304534A1
• Filed: 04/17/2015
• Published: 10/22/2015
• Est. Priority Date: 04/19/2014
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Kadambi, Achuta, Bhandari, Ayush, Raskar, Ramesh
• Granted Patent: US 9,451,141 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4815 using multiple transmitters
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

In illustrative implementations of this invention, an imaging system includes multiple light sources that illuminate a scene, and also includes a lock-in time of flight camera. While the scene is illuminated by these light sources, each of the light sources is amplitude-modulated by a different modulation pattern, and a reference signal is applied to the lock-in time-of-flight camera. The modulation patterns and the reference signal are carefully chosen such that the imaging system is able to disentangle, in real time, the respective contributions of the different light sources, and to compute, in real-time, depth of the scene. In some cases, the modulation signals for the light sources are orthogonal to each other and the reference signal is broadband. In some cases, the modulation codes for the light sources and the reference code are optimal codes that are determined by an optimization algorithm.

Claims

1. A method comprising, in combination:

(a) a set of multiple light sources illuminating a scene;

(b) a signal generator generating a reference signal and a set of multiple modulation signals, such that, while the light sources illuminate the scene

(i) the modulation signals control amplitude of light emitted by the light sources, causing the amplitude of light emitted by each light source to vary over time;

(ii) the modulation signal for each light source is different than the modulation signal for the other light sources, and

(iii) the modulation signals overlap in time and cause two or more of the light sources to simultaneously emit light during multiple time intervals of the modulation signals;

(c) each pixel in the camera

(i) measuring an incident intensity, which incident intensity is intensity of light that reflects from the scene and is incident on the pixel, and

(ii) outputting data indicative of a cross-correlation of the incident intensity and the reference signal; and

(d) a computer taking the data as an input and estimating, for each pixel of the camera

(i) a separate intensity for each respective light source, out of the set of light sources, which separate intensity is intensity of light that (A) is emitted by the respective light source and not by other light sources, (B) then reflects from the scene and (C) then travels to the pixel, and

(ii) depth of a point in the scene that reflects light to the pixel.

2. The method of claim 1, wherein the modulation signals are orthogonal to each other.

3. The method of claim 1, wherein:

(a) the camera has a sampling rate, which sampling rate is equal to the number of raw frames of the camera per second; and

(b) the reference signal is broadband with respect to the sampling rate.

4. The method of claim 1, wherein:

(a) the camera has a rate sampling rate, which sampling rate is equal to the number of raw frames of the camera per second; and

(b) the modulation signals are orthogonal to each other and the reference signal is broadband with respect to the sampling rate.

5. The method of claim 1, wherein data indicative of the modulation signals and reference signal:

(a) is stored in an electronic memory device; and

(b) is read from the memory device and is used by a signal generator that generates the modulation signals and reference signal.

6. The method of claim 5, wherein:

(a) before the data is stored in the electronic memory device, a computer performs an algorithm to calculate the modulation signals and the reference signal; and

(b) the algorithm includes calculating a best linear unbiased estimator that solves Equation 12.

7. The method of claim 5, wherein:

(a) before the data is stored in the electronic memory device, a computer performs an algorithm to calculate the modulation signals and the reference signal; and

(b) the algorithm includes performing semidefinite optimization to solve Equation 20.

8. The method of claim 5, wherein:

(a) before the data is stored in the electronic memory device, a computer performs an algorithm to calculate the modulation signals and the reference signal; and

(b) the algorithm includes performing non-negative matrix factorization to solve Equation 22.

9. The method of claim 1, wherein the camera comprises a lock-in time of flight camera.

10. The method of claim 1, wherein:

(a) the modulation signals are periodic; and

(b) multiple periods of each of the modulation signals occur during each raw frame of the camera.

11. A system comprising, in combination:

(a) a set of multiple light sources for illuminating a scene;

(b) a signal generator programmed to generate a reference signal and a set of multiple modulation signals, such that

(i) the modulation signals control amplitude of light emitted by the light sources, causing the amplitude of light emitted by each light source to vary over time while the light sources illuminate the scene;

(ii) the modulation signal for each light source is different than the modulation signal for the other light sources, and

(iii) the modulation signals overlap in time and cause two or more of the light sources to simultaneously emit light during multiple time intervals of the modulation signals;

(c) a camera configured to output, for each pixel of the camera, data indicative of a cross-correlation of (i) the reference signal and (ii) light reflected from the scene and incident on the pixel; and

(d) a computer programmed to take the data as an input and to estimate, for each pixel of the camera;

(i) an intensity for each respective light source, which intensity for each respective light source is equal to intensity of light that (A) is emitted by the respective light source and not by other light sources, (B) then reflects from the scene and (C) then travels to the pixel; and

(ii) depth of a point in the scene that reflects light to the pixel.

12. The system of claim 11, wherein each of the modulation signals is periodic and has a period that is shorter than the duration of a raw frame of the camera.

13. The system of claim 11, wherein the camera comprises a lock-in time of flight camera.

14. The system of claim 11, wherein the modulation signals are orthogonal to each other.

15. The system of claim 11, wherein:

(a) the camera has a sampling rate, which sampling rate is equal to the number of raw frames of the camera per second; and

(b) the reference signal is broadband with respect to the sampling rate.

16. The system of claim 11, wherein:

(a) the camera has a sampling rate, which sampling rate is equal to the number of raw frames of the camera per second; and

(b) the modulation signals are orthogonal to each other and the reference signal is broadband with respect to the sampling rate.

17. The system of claim 11, wherein the system includes an electronic storage device, which storage device stores data indicative of the modulation signals and reference signals.

18. The system of claim 11, wherein the modulation signals and reference signals are optimal signals according to Equation 20.

19. The system of claim 11, wherein the modulation signals and reference signals are optimal signals according to Equation 23.

20. A system comprising, in combination:

(a) a set of multiple light sources;

(b) a signal generator;

(c) a lock-in time of flight camera;

(d) a computer; and

(e) one or more non-transitory, machine-readable media with instructions encoded thereon

(i) for the signal generator to generate a reference signal and a set of multiple modulation signals, such that

(A) the modulation signals control amplitude of light emitted by the light sources, causing the amplitude of light emitted by each light source to vary over time while the light sources illuminate a scene,

(B) the modulation signal for each light source is different than the modulation signal for the other light sources, and

(C) the modulation signals overlap in time and cause two or more of the light sources to simultaneously emit light during multiple time intervals of the modulation signals; and

(ii) for the computer to take data gathered by the camera as an input and to calculate, for each pixel of the camera

(A) an intensity for each respective light source, out of the set of light sources, which intensity is intensity of light that (I) is emitted by the respective light source and not by other light sources, (II) then reflects from the scene and (III) then travels to the pixel, and

(B) depth of a point in the scene that reflects light to the pixel.