Detecting interference in depth images captured using overlapping depth cameras

• US 10,681,338 B1
• Filed: 09/27/2018
• Issued: 06/09/2020
• Est. Priority Date: 07/24/2018
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Chen, Bo, Pulla, Venkata Sri Krishnakanth, Yan, Junrong
• Primary Examiner(s): Selby, Gevell V
• Application Number:
  US16/144,567
• Time in Patent Office: 621 Days
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Discrete frequencies and time slots of operation are assigned to each of a plurality of time-of-flight cameras. Where two time-of-flight cameras having overlapping fields of view, whether the time-of-flight cameras are operating at the same frequency or time slot is determined by calculating ratios of zero-value pixels to total numbers of pixels for each depth image captured by the time-of-flight cameras over a selected interval. If the time-of-flight cameras operate at the same frequency or time slot, a plot of the ratios of depth images captured using one time-of-flight camera is erratically sinusoidal. Another time-of-flight camera causing the interference may be identified among time-of-flight cameras operating at the frequency or time slot, based on areas of interest that overlap with the time-of-flight camera, or based on a time at which the time-of-flight cameras began capturing depth images, as compared to a time at which the interference is observed.

Claims

1. A facility comprising:

a first time-of-flight camera having a first illuminator and a first sensor, wherein the first time-of-flight camera has a first field of view;

a second time-of-flight camera having a second illuminator and a second sensor, wherein the second time-of-flight camera has a second field of view; and

a server in communication with each of the first time-of-flight camera and the second time-of-flight camera,

wherein the server is configured to execute a method comprising;

programming the first time-of-flight camera to capture depth images by illuminating at least a portion of the facility at a first frequency or during a first time slot;

illuminating, by the first time-of-flight camera, the portion of the facility at the first frequency or during the first time slot;

capturing a plurality of depth images by the first time-of-flight camera at the first frequency or during the first time slot;

calculating, for each of a subset of the plurality of depth images, a ratio of a number of depth image pixels having a zero value to a total number of depth image pixels;

generating a plot based at least in part on the ratios calculated for each of the subset of the plurality of depth images;

determining, based at least in part on the plot, that the first time-of-flight camera is experiencing depth interference; and

in response to determining that the first time-of-flight camera is experiencing depth interference,

programming the second time-of-flight camera to capture depth images by illuminating at least the portion of the scene at a second frequency or during a second time slot,

wherein the second frequency is not the first frequency, and

wherein the second time slot is not the first time slot.

2. The facility of claim 1, wherein the plot is a difference plot comprising differences between ratios calculated for successively captured pairs of the first plurality of depth images on a first axis and times of capture for the first plurality of depth images on a second axis, and

wherein determining that the first time-of-flight camera is experiencing depth interference comprises:

determining that a waveform of the plot corresponds to depth interference.

3. The facility of claim 1, further comprising a working surface having a plurality of areas of interest, and

wherein the method further comprises:

determining that the first field of view includes at least a first area of interest;

identifying a plurality of time-of-flight cameras within the facility having the first area of interest within a field of view; and

selecting one of the plurality of time-of-flight cameras,

wherein the second time-of-flight camera is the selected one of the plurality of time-of-flight cameras, and

wherein the second time-of-flight camera is programmed to capture depth images by illuminating at least the portion of the scene at the second frequency or during the second time slot in response to selecting the one of the plurality of time-of-flight cameras.

4. The facility of claim 1, wherein the method further comprises:

determining that at least one of a plurality of time-of-flight cameras within the facility other than the first time-of-flight camera is programmed to capture depth images by illuminating at least the portion of the scene at the first frequency or during the first time slot; and

selecting one of the plurality of time-of-flight cameras, wherein the second time-of-flight camera is the selected one of the plurality of time-of-flight cameras, and

wherein the second time-of-flight camera is programmed to capture depth images by illuminating at least the portion of the scene at the second frequency or during the second time slot in response to selecting the one of the plurality of time-of-flight cameras.

5. The facility of claim 1, wherein the first time-of-flight camera is configured to capture depth images at a first image rate,

wherein capturing the plurality of depth images comprises:

capturing the plurality of depth images by the first time-of-flight camera at the first image rate; and

selecting the subset of the plurality of depth images, wherein each of the depth images of the subset was captured by the first time-of-flight camera within a selected interval of time.

6. A method comprising:

configuring a first depth camera to capture depth images by illuminating a first field of view at a first frequency;

capturing at least a first plurality of depth images by the first depth camera, wherein capturing the first plurality of depth images comprises illuminating the first field of view at the first frequency, and wherein each of the first plurality of depth images comprises a predetermined number of pixel values;

determining, for each of the first plurality of depth images, a number of pixel values that are approximately equal to zero;

calculating, for each of the first plurality of depth images, a ratio of the number of pixel values that are approximately equal to zero to the predetermined number of pixel values;

determining that at least some of the ratios calculated for each of the first plurality of depth images are consistent with depth interference; and

in response to determining that at least some of the ratios calculated for each of the first plurality of depth images are consistent with depth interference,

storing an indication that the first depth camera is experiencing depth interference in at least one data store.

7. The method of claim 6, wherein the first depth camera is one of a plurality of depth cameras mounted within a facility,

wherein each of the plurality of depth cameras is in communication with a server, and

wherein the method further comprises:

selecting, by the server for each of the plurality of depth cameras, a frequency for illuminating a respective field of view of one of the plurality of depth cameras, wherein the first frequency is the selected frequency for the first depth camera;

configuring, by the server, each of the plurality of depth cameras to capture depth images by illuminating the respective field of view at the frequency selected for the respective depth camera; and

capturing a plurality of depth images by each of the plurality of depth cameras, wherein capturing the plurality of depth images comprises illuminating the respective field of view at the frequency selected for the respective depth camera, wherein each of the plurality of depth images comprises a predetermined number of pixel values, and wherein the first plurality of depth images is the plurality of depth images captured by the first depth camera.

8. The method of claim 6, wherein the first depth camera is configured to capture depth images at a first image rate,

wherein capturing at least the first plurality of depth images comprises:

capturing a second plurality of depth images by the first depth camera at the first image rate; and

selecting the first plurality of depth images from the second plurality of depth images, wherein each of the first plurality of depth images was captured by the first depth camera within a selected interval of time.

9. The method of claim 6, wherein determining that at least some of the ratios calculated for each of the first plurality of depth images are consistent with depth interference comprises:

generating a first plot of the at least some of the ratios; and

determining, based at least in part on the first plot, that the first depth camera is experiencing depth interference.

10. The method of claim 9, wherein the first plot is a difference plot comprising differences between ratios calculated for successively captured pairs of the first plurality of depth images on a first axis and times of capture for the first plurality of depth images on a second axis, and

wherein determining that at least the portion of the ratios calculated for each of the first plurality of depth images are consistent with depth interference further comprises:

determining that the difference plot comprises relative maximum and minimum values at substantially repeating intervals over time.

11. The method of claim 9, wherein determining that the first depth camera is experiencing depth interference comprises:

identifying a second plot of ratios of numbers of pixel values that are approximately equal to zero within a second plurality of depth images to predetermined numbers of pixel values of the second plurality of depth images; and

determining that the first plot is consistent with the second plot.

12. The method of claim 7, further comprising:

in response to determining that at least some of the ratios calculated for each of the first plurality of depth images are consistent with depth interference,

shutting down the first depth camera;

or

configuring the first depth camera to capture depth images by illuminating the first field of view at a second frequency.

13. The method of claim 6, further comprising:

in response to determining that at least some of the ratios calculated for each of the first plurality of depth images are consistent with depth interference,

determining that a second depth camera is configured to capture depth images by illuminating a second field of view at the first frequency.

14. The method of claim 13,

wherein the first field of view overlaps at least in part with the second field of view.

15. The method of claim 7, wherein that at least some of the ratios calculated for each of the first plurality of depth images are consistent with depth interference is determined at a first time, and

wherein the method further comprises:

determining times at which each of the plurality of depth cameras was configured to capture depth images by illuminating the respective field of view; and

identifying a second time, wherein the second time is a latest one of the times prior to the first time; and

selecting one of the plurality of depth cameras based at least in part on the second time.

16. A method comprising:

programming each of a plurality of depth cameras to capture depth images by illuminating at least a portion of a facility at a selected frequency or within a selected time slot, wherein the facility has a plurality of areas of interest, and wherein each of the plurality of depth cameras was programmed at a selected time;

capturing streams of depth images by each of the plurality of depth cameras, wherein each of the plurality of depth cameras has a field of view including at least one of the areas of interest;

selecting a first plurality of depth images of a first stream captured by a first depth camera, wherein the first depth camera is configured to capture depth images by illuminating at least a first area of interest at a first frequency or within a first time slot, and wherein each of the first plurality of depth images is captured during a first interval of time;

determining, for each of the first plurality of depth images, a number of pixels having a value of approximately zero;

calculating, for each of the first plurality of depth images, a ratio of the number of pixels having the value of approximately zero and a total number of pixels;

determining that at least one of the ratios calculated for each of the first plurality of depth images exceeds a predetermined threshold;

in response to determining that the at least one of the ratios calculated for each of the first plurality of depth images exceeds the predetermined threshold,

determining selected frequencies or selected time slots for each of the plurality of depth cameras;

determining the at least one area of interest included within fields of view of each of the plurality of depth cameras; and

determining times at which each of the plurality of depth cameras was configured to capture imaging data by illuminating the at least one area of interest of the scene;

selecting a second depth camera based at least in part on at least one of the selected frequencies, the selected time slots, the at least one area of interest or the times; and

programming the second depth camera to capture depth images by illuminating the at least one area of interest at a second frequency or a second time slot,

wherein the second frequency is not the first frequency, and

wherein the second time slot is not the first time slot.

17. The method of claim 16, wherein the second depth camera is one of the plurality of depth cameras programmed to capture depth images by illuminating the at least one area of interest at the first frequency or within the first time slot.

18. The method of claim 16, wherein a second field of view of the second depth camera includes the first area of interest.

19. The method of claim 16, wherein that the at least one of the ratios calculated for each of the first plurality of depth images is determined to exceed a predetermined threshold at a first time,

wherein the second depth camera was programmed to capture depth images at a selected frequency at a second time, and

wherein the second time is a latest one of the times prior to the first time.

20. The method of claim 16, wherein determining that the at least one of the ratios calculated for each of the first plurality of depth images exceeds the predetermined threshold comprises:

generating a difference plot comprising differences between ratios calculated for successively captured pairs of the first plurality of depth images on a first axis and times of capture for the first plurality of depth images on a second axis; and

determining, based at least in part on the difference plot, that the first depth camera is experiencing depth interference.