Methods and apparatus for multi-frequency camera

• US 9,405,008 B2
• Filed: 05/16/2014
• Issued: 08/02/2016
• Est. Priority Date: 05/17/2013
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Raskar, Ramesh, Kadambi, Achuta, Bhandari, Ayush, Barsi, Christopher
• Primary Examiner(s): Diep, Trung
• Application Number:
  US14/280,284
• Publication Number:
  US 20140340569A1
• Time in Patent Office: 809 Days
• Field of Search: 348/96, 348296-297, 348/280, 348/283, 348/505, 359/474, 359/483
• US Class Current: 1/1
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/493 Extracting wanted echo signals

Abstract

In exemplary implementations of this invention, a multi-frequency ToF camera mitigates the effect of multi-path interference (MPI), and can calculate an accurate depth map despite MPI. A light source in the multi-frequency camera emits light in a temporal sequence of different frequencies. For example, the light source can emit a sequence of ten equidistant frequencies f=10 MHz, 20 MHz, 30 MHz, . . . , 100 MHz. At each frequency, a lock-in sensor within the ToF camera captures 4 frames. From these 4 frames, one or more processors compute, for each pixel in the sensor, a single complex number. The processors stack all of such complex quantities (one such complex number per pixel per frequency) and solve for the depth and intensity, using a spectral estimation technique.

Claims

1. A method comprising, in combination:

(a) using a light source to emit light in a temporal sequence of frequencies, such that

(i) the light illuminates a scene,

(ii) reflected light from the scene reaches a light sensor, and

(iii) a first frequency is emitted during a first period of time in the sequence and not during other periods of time during the sequence, a second frequency is emitted during a second period of time in the sequence and not during other periods of time during the sequence, and a third frequency is emitted during a third period of time in the sequence and not during other periods of time during the sequence;

(b) using a light sensor to capture multiple frames for each respective frequency in the sequence; and

(c) using one or more processors

(i) to analyze data in the frames and

(ii) to calculate a set of complex numbers for each respective pixel in the light sensor, such that the set of complex numbers includes a complex number for each respective frequency in the sequence for each respective path length in a set of path lengths, wherein the set of path lengths consists of path lengths of paths of light that end at the respective pixel.

2. The method of claim 1, wherein the method further comprises using the one or more processors to calculate a depth map of the scene.

3. The method of claim 1, wherein the paths of light include a single-bounce path.

4. The method of claim 1, wherein the paths of light include a multiple-bounce path.

5. The method of claim 1, wherein the light sensor comprises a lock-in sensor.

6. The method of claim 1, wherein the light sensor captures four frames for each respective frequency in the sequence of frequencies.

7. The method of claim 1, wherein the one or more processors calculate multiple components of light incident on a single pixel of the light sensor, each respective component, out of the multiple components, comprising light that travels a particular path length, out of the set of path lengths.

8. The method of claim 1, wherein the algorithm includes steps for computing frequency components of a signal.

9. The method of claim 1, wherein the algorithm includes a matrix pencil computation.

10. The method of claim 1, wherein the algorithm includes a computation that uses Prony'"'"'s Method.

11. The method of claim 1, wherein the algorithm includes a computation that uses a MUSIC method of spectral estimation.

12. The method of claim 1, wherein the algorithm includes a computation that uses a maximum likelihood method of spectral estimation.

13. The method of claim 1, wherein the algorithm includes a computation that uses an ESPRIT method of spectral estimation.

14. The method of claim 1, wherein the algorithm includes a computation that uses a parametric method of spectral estimation.

15. A method of determining a depth map of a scene using a time-of-flight camera, which camera includes a light sensor,

THE IMPROVEMENT COMPRISING:

(a) using a light source to emit light in a temporal sequence of frequencies to illuminate the scene, such that a first frequency is emitted during a first period of time in the sequence and not during other periods of time during the sequence, a second frequency is emitted during a second period of time in the sequence and not during other periods of time during the sequence, and a third frequency is emitted during a third period of time in the sequence and not during other periods of time during the sequence; and

(b) using one or more processors to estimate a phase and coefficient of reflection for each respective pixel of the light sensor for each respective path length, out of a set of path lengths of light, each respective path length being for a path that starts at the light source and ends at the light sensor.

16. A time-of-flight camera, which camera includes a light sensor,

THE IMPROVEMENT COMPRISING:

(a) one or more light sources for emitting light in a temporal sequence of different frequencies, such that a first frequency is emitted during a first period of time in the sequence and not during other periods of time during the sequence, a second frequency is emitted during a second period of time in the sequence and not during other periods of time during the sequence, and a third frequency is emitted during a third period of time in the sequence and not during other periods of time during the sequence; and

(b) one or more processors programmed to estimate a phase and coefficient of reflection for each respective pixel of the light sensor for each respective path length, out of a set of path lengths of light, each respective path length being for a path that starts at the light source and ends at the light sensor.

17. The time-of-flight camera of claim 16, wherein the one or more processors are further programmed to calculate a depth map of a scene.

18. The time-of-flight camera of claim 16, wherein the one or more processors are further programmed to identify a direct component of light incident on a pixel of the sensor.

19. The time-of-flight camera of claim 16, wherein the one or more processors are further programmed to identify a global component of light incident on a pixel of the sensor.