Detecting interference between time-of-flight cameras using modified image sensor arrays

• US 10,708,484 B2
• Filed: 11/05/2018
• Issued: 07/07/2020
• Est. Priority Date: 06/20/2018
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Becker, Gabor
• Primary Examiner(s): Duley, Janese
• Application Number:
  US16/180,474
• Publication Number:
  US 20190394404A1
• Time in Patent Office: 610 Days
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/662 by using master/slave camera arrangements for affecting the control of camera image capture, e.g. placing the camera in a desirable condition to capture a desired image

Abstract

Time-of-flight cameras may be synchronized where the fields of view of the time-of-flight cameras overlap. The time-of-flight cameras may be programmed within intervals of time for illuminating their respective fields of view that do not conflict with one another. When a first time-of-flight camera illuminates a first field of view that overlaps with a second field of view of a second time-of-flight camera, and the second time-of-flight camera detects reflected light from the illumination, the second time-of-flight camera may determine a time to illuminate the second field of view based on the reflected light. A time-of-flight camera may include an image sensor modified to include one or more modulated light sensors among an array of photoreceptors. A modulation frequency or illumination interval may be selected for the time-of-flight camera based on modulation frequencies or illumination intervals of other cameras, as determined based on data captured by the modulated light sensors.

Claims

1. A working environment comprising:

a first time-of-flight camera having a first illuminator and a first sensor, wherein the first time-of-flight camera has a first field of view;

a second time-of-flight camera having a second illuminator and a second sensor, wherein the second time-of-flight camera has a second field of view, wherein the second sensor comprises a second plurality of modulated light sensors and a second plurality of photoreceptors arranged in an array, and wherein the second field of view overlaps the first field of view; and

a server in communication with each of the first time-of-flight camera and the second time-of-flight camera,

wherein the server is configured to execute a method comprising;

causing the first time-of-flight camera to illuminate at least a first portion of a scene within the first field of view with first modulated light by the first illuminator at a first illumination interval within a repeated interval;

determining that at least one of the second plurality of modulated light sensors was exposed to the first modulated light during the first illumination interval;

in response to determining that the at least one of the second plurality of modulated light sensors was exposed to the first modulated light during the first illumination interval,

determining a first modulation frequency of the first modulated light;

selecting a second modulation frequency based at least in part on the first modulation frequency;

selecting a second illumination interval within the repeated interval, wherein the second illumination interval is not the first illumination interval; and

causing the second time-of-flight camera to illuminate at least the second portion of the scene within the second field of view with second modulated light by the second illuminator at a second time,

wherein the second time is within the second illumination interval.

2. The working environment of claim 1, wherein the second sensor is one of a complementary metal-oxide sensor or a charge coupled device modified to include the second plurality of modulated light sensors.

3. The working environment of claim 1, wherein each of the second plurality of modulated light sensors is coupled to a bandpass filter circuit comprising at least one operational amplifier, at least one filtering capacitor and at least one filtering resistor,

wherein at least one input to the at least one operational amplifier is proportional to a current flowing across a photodiode in response to the first modulated light, and

wherein the first modulation frequency is determined based at least in part on an output of the at least one of the operational amplifier.

4. The working environment of claim 1, wherein the repeated interval is one second,

wherein the first time is within a first slot of the repeated interval, and

wherein the second time is within a second slot of the repeated interval.

5. A method comprising:

capturing, by a first sensor of a first imaging device, first modulated light reflected from at least a portion of a scene within a first field of view of the first imaging device, wherein the first imaging device further comprises a first illuminator, and wherein the first sensor comprises at least one photoreceptor and at least one modulated light sensor;

programming the first imaging device with a data record defining a repeated interval and a plurality of slots within the repeated interval, wherein the first imaging device is programmed prior to a first time;

determining, by at least one computer processor, at least one of a first modulation frequency of the first modulated light or a first illumination interval of the first modulated light at the first time, wherein the first illumination interval is a first one of the plurality of slots;

selecting a second one of the plurality of slots based at least in part on the at least one of the first modulation frequency or the first illumination interval, wherein the second one of the plurality of slots comprises a second illumination interval, and wherein the second one of the plurality of slots is not the first one of the plurality of slots; and

illuminating, by the first illuminator at a second time, the first field of view with second modulated light, wherein the second modulated light has a second modulation frequency or is illuminated at the second illumination interval within the repeated interval,

wherein the second time follows the first time.

6. The method of claim 5, wherein the first sensor comprises an array having a plurality of locations,

wherein the array further comprises the at least one modulated light sensor in at least one of the plurality of locations, and

wherein the array further comprises a plurality of photoreceptors distributed in locations other than the at least one location having the at least one modulated light sensor.

7. The method of claim 6, wherein the first sensor comprises a plurality of modulated light sensors, and

wherein the plurality of modulated light sensors are distributed throughout the array at a regular interval along a first axis and at a regular interval along a second axis.

8. The method of claim 6, further comprising:

capturing at least some of the first modulated light by each of the plurality of photoreceptors;

determining times at which the at least some of the first modulated light is received by each of the plurality of photoreceptors;

determining, for each of the plurality of photoreceptors, a difference between the second time and one of the times; and

generating a depth image based at least in part on the differences.

9. The method of claim 6, wherein the first sensor is one of a complementary metal-oxide sensor or a charge coupled device modified to replace at least one of the plurality of photoreceptors with the at least one modulated light sensor.

10. The method of claim 5, wherein the at least one modulated light sensor comprises a lens assembly for mapping an area of sensitivity of the modulated light sensor or one or more photodiodes therein to the first field of view.

11. The method of claim 10, wherein the at least one modulated light sensor further comprises a circuit comprising at least one operational amplifier, at least one filtering capacitor and at least one filtering resistor, and

wherein the first modulation frequency is determined based at least in part on an output of the operational amplifier.

12. The method of claim 5, wherein the first imaging device is mounted to a first vehicle, and

wherein the first vehicle further comprises the at least one computer processor.

13. The method of claim 12, further comprising:

determining, based at least in part on the at least one of the first modulation frequency or the first illumination interval, that a second imaging device mounted to a second vehicle illuminated at least the portion of the scene with the first modulated light during the first illumination interval.

14. The method of claim 5, wherein a first wavelength of the first modulated light is within a range of approximately seven hundred nanometers to one thousand nanometers.

15. A time-of-flight camera comprising:

an illuminator;

sensor comprising a plurality of modulated light sensors and a first plurality of photoreceptors arranged in an array; and

a computer device comprising at least one processor and at least one memory component,

wherein the time-of-flight camera defines a first field of view, and

wherein the computer device is configured to execute a method comprising;

identifying a data record including a plurality of time slots at a repeated interval;

capturing, by at least one of the first plurality of modulated light sensors, first modulated light at a first time;

determining a first modulation frequency of the first modulated light;

determining a first illumination interval of the first modulated light;

determining that the first illumination interval corresponds to a first one of the plurality of time slots;

selecting a second one of the plurality of time slots, wherein the second one of the plurality of time slots is not the first one of the plurality of time slots;

selecting a second modulation frequency based at least in part on the first modulation frequency;

selecting a time for illuminating the first field of view by the illuminator based at least in part on the first illumination interval, wherein the time is within the second one of the plurality of time slots; and

causing the first illuminator to illuminate the first field of view with second modulated light at the time,

wherein the second modulated light has the second modulation frequency.

16. The first time-of-flight camera of claim 15, further comprising an amplifier circuit,

wherein at least one of the first plurality of modulated light sensors is coupled to an input to the circuit, and

wherein the first modulation frequency is determined based at least in part on an output from the circuit.

17. The first time-of-flight camera of claim 15, wherein the first time-of-flight camera is mounted to one of an aerial vehicle, an autonomous ground vehicle or an autonomous mobile robot.

18. The first time-of-flight camera of claim 15, further comprising:

capturing, by each of the first plurality of photoreceptors, at least some of the second modulated light reflected from a scene during the second illumination interval;

determining times at which each of the first plurality of photoreceptors captured the at least some of the second modulated light during the second illumination interval; and

generating a depth image of the scene based at least in part on differences between the time and each of the times at which each of the first plurality of photoreceptors captured the reflected light during the second illumination interval.

19. The time-of-flight camera of claim 16, wherein the sensor is one of a complementary metal-oxide sensor or a charge coupled device modified to include the plurality of modulated light sensors.

20. The time-of-flight camera of claim 15, wherein each of the plurality of modulated light sensors is coupled to a bandpass filter circuit comprising at least one operational amplifier, at least one filtering capacitor and at least one filtering resistor,

wherein at least one input to the at least one operational amplifier is proportional to a current flowing across a photodiode in response to the first modulated light, and

wherein the first modulation frequency is determined based at least in part on an output of the at least one operational amplifier.