Synchronizing time-of-flight cameras

• US 10,674,063 B2
• Filed: 06/20/2018
• Issued: 06/02/2020
• Est. Priority Date: 06/20/2018
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Becker, Gabor Szedo
• Primary Examiner(s): Nazrul, Shahbaz
• Application Number:
  US16/013,760
• Publication Number:
  US 20190394386A1
• Time in Patent Office: 713 Days
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/28 Mobile studios

Abstract

Time-of-flight cameras may be synchronized where the fields of view of the time-of-flight cameras overlap. The time-of-flight cameras may be programmed within intervals of time for illuminating their respective fields of view that do not conflict with one another. When a first time-of-flight camera illuminates a first field of view that overlaps with a second field of view of a second time-of-flight camera, and the second time-of-flight camera detects reflected light from the illumination, the second time-of-flight camera may determine a time to illuminate the second field of view based on the reflected light. In this manner, any number of time-of-flight cameras may be synchronized with one another without requiring a direct connection between the time-of-flight cameras.

Claims

1. A working environment comprising:

a first time-of-flight camera having a first illuminator and a first sensor, wherein the first time-of-flight camera has a first field of view;

a second time-of-flight camera having a second illuminator and a second sensor, wherein the second time-of-flight camera has a second field of view, wherein the second field of view overlaps the first field of view; and

a server in communication with each of the first time-of-flight camera and the second time-of-flight camera,

wherein the server is configured to execute a method comprising;

transmitting a data record to each of the first time-of-flight camera and the second time-of-flight camera, wherein the data record identifies a first time with respect to an interval for illuminating at least a first portion of a scene at the working environment corresponding to the first field of view by the first time-of-flight camera and a second time within a repeated interval for illuminating at least a second portion of the scene corresponding to the second field of view by the second time-of-flight camera, and wherein the second time is defined with respect to the interval;

causing the first time-of-flight camera to illuminate at least the first portion of the scene corresponding to the first field of view by the first illuminator at a third time based at least in part on the data record;

determining that the second sensor of the second time-of-flight camera captured light reflected from at least the first portion of the scene;

in response to determining that the second sensor of the second time-of-flight camera captured light reflected from at least the first portion of the scene,

selecting a fourth time for illuminating at least a second portion of the scene corresponding to the second field of view by the second illuminator based at least in part on the data record; and

causing the second time-of-flight camera to illuminate at least the second portion of the scene corresponding to the second field of view by the second illuminator at the fourth time.

2. The working environment of claim 1, wherein causing the first time-of-flight camera to illuminate at least the first portion of the scene comprises:

causing the first illuminator to emit at least a first pulse of modulated light at one of a first infrared frequency or a first infrared wavelength at the third time, and

wherein causing the second time-of-flight camera to illuminate at least the second portion of the scene comprises;

causing the second illuminator to emit at least a second pulse of modulated light at one of a second infrared frequency or a second infrared wavelength at the fourth time.

3. The working environment of claim 1, wherein the second sensor comprises an array of photoreceptors, and

wherein determining that the second sensor of the second time-of-flight camera captured light reflected from at least the first portion of the scene comprises:

identifying a first plurality of photoreceptors of the array corresponding to the first portion of the scene; and

determining that at least some of the first plurality of photoreceptors detected the first pulse of modulated light at the one of the first infrared frequency or the first infrared wavelength.

4. The working environment of claim 1, wherein the repeated interval is one second,

wherein the first time is within a first slot of the repeated interval, and

wherein the second time is within a second slot of the repeated interval.

5. A method comprising:

illuminating at least a portion of a scene by a first illuminator of a first imaging device, wherein the first imaging device further comprises a first sensor, and wherein the portion of the scene is within a first field of view of the first imaging device and a second field of view of a second imaging device;

capturing light reflected from the portion of the scene by a second sensor of the second imaging device at a first time, wherein the second imaging device further comprises a second illuminator;

selecting, by the second imaging device, a second time for illuminating the portion of the scene by the second illuminator based at least in part on capturing the light at the first time; and

illuminating at least the portion of the scene by the second illuminator at the second time.

6. The method of claim 5, further comprising:

programming the first imaging device with a data record defining a repeated interval of time including a first illumination interval for the first imaging device within the repeated interval and a second illumination interval for the second imaging device within the repeated interval; and

programming the second imaging device with the data record,

wherein at least the portion of the scene is illuminated by the first illuminator during the first illumination interval, and

wherein the second time is selected based at least in part on the first time and the second illumination interval.

7. The method of claim 5, further comprising:

wherein the data record further defines a plurality of slots within the repeated interval,

wherein the first illumination interval is a first one of the slots, and

wherein the second illumination interval is a second one of the slots.

8. The method of claim 5, wherein illuminating at least the portion of the scene by the first illuminator comprises:

emitting at least a first modulated light pulse by the first illuminator for a first period of time that includes the first time, and

wherein illuminating at least the portion of the scene by the second illuminator comprises;

emitting at least a second modulated light pulse by the second illuminator for a second period of time that includes the second time.

9. The method of claim 8, wherein a first wavelength of the first modulated light pulse is within a range of approximately seven hundred nanometers to one thousand nanometers.

10. The method of claim 5, wherein the second sensor comprises a plurality of photoreceptors arranged in an array, and

wherein capturing the light reflected from the portion of the scene by the second sensor comprises:

detecting the light reflected from the portion of the scene by at least some of the plurality of photoreceptors of the second sensor at the first time.

11. The method of claim 10, wherein the second sensor is one of a complementary metal-oxide sensor or a charge coupled device.

12. The method of claim 10, further comprising:

prior to the first time,

identifying a portion of the second sensor corresponding to the portion of the scene, wherein the portion of the second sensor comprises the at least some of the plurality of photoreceptors of the second sensor; and

exposing each of the at least some of the photoreceptors for an interval including the first time.

13. The method of claim 10, further comprising:

capturing light reflected from at least the portion of the scene by each of the plurality of photoreceptors;

determining times at which light is received by each of the plurality of photoreceptors;

determining, for each of the plurality of photoreceptors, a difference between the second time and one of the times; and

generating a depth image based at least in part on the differences.

14. The method of claim 5, wherein the portion of the scene corresponds to a region defined by an overlap of a first portion of the first field of view and a second portion of the second field of view.

15. The method of claim 5, wherein the first imaging device is mounted to a first vehicle, and

wherein the second imaging device is mounted to one of the first vehicle or a second vehicle.

16. A first aerial vehicle comprising:

a first time-of-flight camera comprising a first illuminator and a first sensor, wherein the first time-of-flight camera defines a first field of view, and wherein the first sensor comprises a first plurality of photoreceptors arranged in an array; and

a first computer device comprising at least one processor and at least one memory component,

wherein the first computer device is configured to execute a method comprising;

receiving a data record over a network, wherein the data record defines a repeated interval having a first illumination interval for the first imaging device and a second illumination interval for a second time-of-flight camera having a second field of view;

storing the data record in the at least one memory component;

capturing, by at least some of the first plurality of photoreceptors, reflected light during the second illumination interval;

selecting a first time for illuminating the first field of view by the first illuminator based at least in part on the captured reflected light and the data record, wherein the first time is within the first illumination interval; and

causing the first illuminator to illuminate the first field of view at the time.

17. The first aerial vehicle of claim 16, wherein the second time-of-flight camera is mounted to one of:

the first aerial vehicle;

a second aerial vehicle;

or

an autonomous mobile robot.

18. The first aerial vehicle of claim 16, further comprising:

identifying a portion of the first plurality of photoreceptors corresponding to an overlap of the first field of view and the second field of view; and

exposing at least the portion of the first plurality of photoreceptors corresponding to the overlap during the second illumination interval.

19. The first aerial vehicle of claim 16, wherein the data record further identifies at least one of a frequency or a wavelength of light associated with a second illuminator of the second time-of-flight camera, and

wherein capturing the reflected light during the second illumination interval comprises:

determining that the reflected light is of the at least one of the frequency or the wavelength,

wherein the time for illuminating the first field of view by the first illuminator is selected in response to determining that the reflected light is of the at least one of the frequency or the wavelength.

20. The first aerial vehicle of claim 19, further comprising:

capturing, by each of the first plurality of photoreceptors, reflected light during the first illumination interval;

determining times at which each of the first plurality of photoreceptors captured the reflected light during the first illumination interval; and

generating a depth image based at least in part on differences between the first time and each of the times at which each of the first plurality of photoreceptors captured the reflected light during the first illumination interval.